[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:28 dist-test-slave-9zdj krb5kdc[4360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:28 dist-test-slave-9zdj krb5kdc[4360](info): set up 2 sockets
Aug 10 15:20:28 dist-test-slave-9zdj krb5kdc[4360](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 15:20:29.677522 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:29 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:29.819305 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37909
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:37909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:29.919128 4376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:29.919396 4376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:29.919425 4376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:29.920787 4376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:29.920837 4376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:29.920853 4376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:29.920867 4376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:29.920881 4376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:29.922761 4376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:37909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:29.923004 4376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:29.923233 4376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:29.929217 4385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:29.929258 4382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:29.929219 4383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:29.929587 4376 server_base.cc:1034] running on GCE node
I20260810 15:20:29.930050 4376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:29.930495 4376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:29.931676 4376 hybrid_clock.cc:648] HybridClock initialized: now 1786375229931622 us; error 78 us; skew 500 ppm
Aug 10 15:20:29 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375229, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:29.934557 4376 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:29.935192 4376 webserver.cc:460] Webserver started at http://127.4.64.126:42495/ using document root <none> and password file <none>
I20260810 15:20:29.935448 4376 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:29.935523 4376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:29.935683 4376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:29.937126 4376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "16be245e90ff4e02b711051c025e6b9d"
format_stamp: "Formatted at 2026-08-10 15:20:29 on dist-test-slave-9zdj"
server_key: "545a0bdc63266b576d70f82a7062d002"
server_key_iv: "202f62c26af114f9e84d193bf2e6e887"
server_key_version: "encryptionkey@0"
I20260810 15:20:29.937678 4376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "16be245e90ff4e02b711051c025e6b9d"
format_stamp: "Formatted at 2026-08-10 15:20:29 on dist-test-slave-9zdj"
server_key: "545a0bdc63266b576d70f82a7062d002"
server_key_iv: "202f62c26af114f9e84d193bf2e6e887"
server_key_version: "encryptionkey@0"
I20260810 15:20:29.939343 4376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:29.940276 4393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:29.940582 4376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:29.940704 4376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "16be245e90ff4e02b711051c025e6b9d"
format_stamp: "Formatted at 2026-08-10 15:20:29 on dist-test-slave-9zdj"
server_key: "545a0bdc63266b576d70f82a7062d002"
server_key_iv: "202f62c26af114f9e84d193bf2e6e887"
server_key_version: "encryptionkey@0"
I20260810 15:20:29.940784 4376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:29.958360 4376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:29.960904 4376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:29.961119 4376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:29.965936 4376 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:37909
I20260810 15:20:29.966009 4453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:37909 every 8 connection(s)
I20260810 15:20:29.966483 4376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:29.967581 4456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:29.970100 4456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d: Bootstrap starting.
I20260810 15:20:29.970799 4456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:29.971130 4456 log.cc:824] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:29.971982 4456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d: No bootstrap required, opened a new log
I20260810 15:20:29.973734 4456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be245e90ff4e02b711051c025e6b9d" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37909 } }
I20260810 15:20:29.973897 4456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:29.973920 4456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16be245e90ff4e02b711051c025e6b9d, State: Initialized, Role: FOLLOWER
I20260810 15:20:29.974099 4456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be245e90ff4e02b711051c025e6b9d" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37909 } }
I20260810 15:20:29.974252 4456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:29.974313 4456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:29.974372 4456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:29.974993 4456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be245e90ff4e02b711051c025e6b9d" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37909 } }
I20260810 15:20:29.975090 4456 leader_election.cc:302] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16be245e90ff4e02b711051c025e6b9d; no voters:
I20260810 15:20:29.975288 4456 leader_election.cc:288] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:29.975389 4459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:29.975571 4456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:29.975584 4459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [term 1 LEADER]: Becoming Leader. State: Replica: 16be245e90ff4e02b711051c025e6b9d, State: Running, Role: LEADER
I20260810 15:20:29.975750 4459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be245e90ff4e02b711051c025e6b9d" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37909 } }
I20260810 15:20:29.976331 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 4376
I20260810 15:20:29.976456 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:29.976434 4461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16be245e90ff4e02b711051c025e6b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be245e90ff4e02b711051c025e6b9d" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37909 } } }
I20260810 15:20:29.976522 4461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:29.976959 4462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16be245e90ff4e02b711051c025e6b9d. Latest consensus state: current_term: 1 leader_uuid: "16be245e90ff4e02b711051c025e6b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be245e90ff4e02b711051c025e6b9d" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37909 } } }
I20260810 15:20:29.977043 4462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:29.977324 4353 external_mini_cluster.cc:1428] Setting key 7e7021f6490c417d475ad2005a48fa28
W20260810 15:20:29.977851 4480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:29.977967 4480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:29.978035 4472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:29.978209 4472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:29.979827 4472 catalog_manager.cc:1347] Generated new cluster ID: 67358c8cbf5241cb807b61594364787a
I20260810 15:20:29.979885 4472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:29.989056 4472 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:29.989851 4472 catalog_manager.cc:1504] Loading token signing keys...
Aug 10 15:20:29 dist-test-slave-9zdj krb5kdc[4360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:29.992606 4472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16be245e90ff4e02b711051c025e6b9d: Generated new TSK 0
I20260810 15:20:29.992795 4472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:29.997390 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:29.978858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.4.64.126:37909)
0810 15:20:29.979054 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:29.979064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:29.979106 (+ 42us) server_negotiation.cc:407] Connection header received
0810 15:20:29.981517 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:29.981527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:29.981593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:29.981793 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:29.984431 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:29.985938 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:29.987753 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:29.988413 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:29.988560 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:29.993348 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:29.993352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:29.993353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:29.993363 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:29.995049 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:29.995631 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:29.995634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:29.995635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:29.995740 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:29.996079 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:29.996080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:29.996080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:29.996360 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:29.996460 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:29.996858 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 15:20:29.996990 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:30.033052 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37909
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:30.122819 4488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:30.123086 4488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:30.123131 4488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:30.124402 4488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:30.124478 4488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:30.124526 4488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:30.126224 4488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:30.126541 4488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:30.126804 4488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:30.133571 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:30.133535 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:30.133535 4498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:30.133872 4488 server_base.cc:1034] running on GCE node
I20260810 15:20:30.134092 4488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:30.134495 4488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:30.135655 4488 hybrid_clock.cc:648] HybridClock initialized: now 1786375230135633 us; error 41 us; skew 500 ppm
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:30.138797 4488 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:30.139362 4488 webserver.cc:460] Webserver started at http://127.4.64.65:43553/ using document root <none> and password file <none>
I20260810 15:20:30.139678 4488 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:30.139771 4488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:30.139943 4488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:30.141160 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "1c398f79002f45b68308d3a3049c70f8"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "d3be64414cfd165faeb08767d9339976"
server_key_iv: "0e318967116591f012a0e72d9e23e5ec"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.141577 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "1c398f79002f45b68308d3a3049c70f8"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "d3be64414cfd165faeb08767d9339976"
server_key_iv: "0e318967116591f012a0e72d9e23e5ec"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.143330 4488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:30.144373 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.144724 4488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:30.144886 4488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "1c398f79002f45b68308d3a3049c70f8"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "d3be64414cfd165faeb08767d9339976"
server_key_iv: "0e318967116591f012a0e72d9e23e5ec"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.144985 4488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:30.170868 4488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:30.173504 4488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:30.173719 4488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:30.174019 4488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:30.174392 4488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:30.174455 4488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.174501 4488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:30.174525 4488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.181391 4488 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:38667
I20260810 15:20:30.181450 4633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:38667 every 8 connection(s)
I20260810 15:20:30.181897 4488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:30.190701 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 4488
I20260810 15:20:30.190846 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:30.191010 4353 external_mini_cluster.cc:1428] Setting key f9944e6b66d73c75849aad4df319b35c
I20260810 15:20:30.193027 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.182761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:50623 (local address 127.4.64.126:37909)
0810 15:20:30.182836 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:30.182839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:30.183057 (+ 218us) server_negotiation.cc:407] Connection header received
0810 15:20:30.183667 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:30.183669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:30.183707 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:30.183778 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:30.185021 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.185602 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.186711 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.186962 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.187038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.189562 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:30.189563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:30.189563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:30.189571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:30.191416 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.191875 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.191877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.191877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.191947 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.192307 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.192308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.192308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.192446 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:30.192525 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:30.192822 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 15:20:30.192932 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:20:30.193252 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.182734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37909 (local address 127.4.64.65:50623)
0810 15:20:30.182924 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:30.182972 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:30.183512 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:30.183877 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:30.183881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:30.184677 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:30.184894 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:30.184899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.185737 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.185738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:30.186585 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:30.186587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.187056 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.187057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:30.187186 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.187755 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:30.187766 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:30.189460 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:30.191511 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:30.191520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:30.191523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:30.191777 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:30.192015 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:30.192016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:30.192017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:30.192225 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:30.192516 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:30.192523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:30.192681 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 15:20:30.192799 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 15:20:30.192985 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260810 15:20:30.194051 4634 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37909
I20260810 15:20:30.194172 4634 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:30.194391 4634 heartbeater.cc:511] Master 127.4.64.126:37909 requested a full tablet report, sending...
I20260810 15:20:30.195043 4414 ts_manager.cc:194] Registered new tserver with Master: 1c398f79002f45b68308d3a3049c70f8 (127.4.64.65:38667)
I20260810 15:20:30.195802 4414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:50623
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:30.221662 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37909
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:30.308634 4641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:30.308862 4641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:30.308892 4641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:30.310017 4641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:30.310056 4641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:30.310083 4641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:30.311568 4641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:30.311784 4641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:30.311977 4641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:30.317502 4647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:30.317528 4648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:30.317502 4650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:30.317814 4641 server_base.cc:1034] running on GCE node
I20260810 15:20:30.318053 4641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:30.318274 4641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:30.319434 4641 hybrid_clock.cc:648] HybridClock initialized: now 1786375230319417 us; error 31 us; skew 500 ppm
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:30.322330 4641 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:30.322796 4641 webserver.cc:460] Webserver started at http://127.4.64.66:40511/ using document root <none> and password file <none>
I20260810 15:20:30.322992 4641 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:30.323033 4641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:30.323124 4641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:30.324196 4641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "0863fa8a137a47f8b3249a64da98fde8"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "b6ffb22b41620372e7352e3c35b5dfc6"
server_key_iv: "3942b67df00ce5bedfa14f34811ca735"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.324523 4641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "0863fa8a137a47f8b3249a64da98fde8"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "b6ffb22b41620372e7352e3c35b5dfc6"
server_key_iv: "3942b67df00ce5bedfa14f34811ca735"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.326099 4641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:30.326912 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.327117 4641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:30.327332 4641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "0863fa8a137a47f8b3249a64da98fde8"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "b6ffb22b41620372e7352e3c35b5dfc6"
server_key_iv: "3942b67df00ce5bedfa14f34811ca735"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.327422 4641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:30.343470 4641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:30.346195 4641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:30.346379 4641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:30.346652 4641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:30.346997 4641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:30.347054 4641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.347101 4641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:30.347124 4641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.353423 4641 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:41859
I20260810 15:20:30.353514 4784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:41859 every 8 connection(s)
I20260810 15:20:30.353879 4641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:30.358139 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 4641
I20260810 15:20:30.358364 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:30.358553 4353 external_mini_cluster.cc:1428] Setting key 9cd598016b482958cd1f04161f9ff5ec
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:30.365345 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.354753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:43017 (local address 127.4.64.126:37909)
0810 15:20:30.354844 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:30.354847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:30.355027 (+ 180us) server_negotiation.cc:407] Connection header received
0810 15:20:30.355547 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:30.355548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:30.355559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:30.355613 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:30.356593 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.357108 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.358304 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.358622 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.358742 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.361786 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:30.361788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:30.361788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:30.361804 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:30.363471 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.363969 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.363971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.363971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.364043 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.364522 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.364523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.364524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.364691 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:30.364777 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:30.365137 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 15:20:30.365242 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 15:20:30.365573 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.354703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37909 (local address 127.4.64.66:43017)
0810 15:20:30.354899 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:30.354938 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:30.355455 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:30.355668 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:30.355671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:30.356295 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:30.356498 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:30.356503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.357278 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.357279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:30.358189 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:30.358192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.358830 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.358832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:30.359005 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.359795 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:30.359812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:30.361598 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:30.363586 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:30.363596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:30.363599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:30.363861 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:30.364123 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:30.364124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:30.364125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:30.364285 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:30.364769 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:30.364778 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:30.364959 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 15:20:30.365090 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 15:20:30.365298 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260810 15:20:30.366474 4785 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37909
I20260810 15:20:30.366577 4785 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:30.366839 4785 heartbeater.cc:511] Master 127.4.64.126:37909 requested a full tablet report, sending...
I20260810 15:20:30.367406 4414 ts_manager.cc:194] Registered new tserver with Master: 0863fa8a137a47f8b3249a64da98fde8 (127.4.64.66:41859)
I20260810 15:20:30.367834 4414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:43017
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:30.391083 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37909
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:30.481240 4792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:30.481526 4792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:30.481571 4792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:30.482815 4792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:30.482888 4792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:30.482936 4792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:30.484467 4792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37909
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:30.484764 4792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:30.485014 4792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:30.490602 4799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:30.490593 4800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:30.490670 4792 server_base.cc:1034] running on GCE node
W20260810 15:20:30.490609 4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:30.491053 4792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:30.491344 4792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:30.492517 4792 hybrid_clock.cc:648] HybridClock initialized: now 1786375230492515 us; error 43 us; skew 500 ppm
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:30.495231 4792 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:30.495756 4792 webserver.cc:460] Webserver started at http://127.4.64.67:46381/ using document root <none> and password file <none>
I20260810 15:20:30.496016 4792 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:30.496088 4792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:30.496243 4792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:30.497432 4792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "5bfd2ad7e5b845ae9558ddc2f1a75063"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "d96badc1fdb74ad86d2f746508681e0a"
server_key_iv: "c7063a27e9f14973e60ec4e9472d9ca5"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.497835 4792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "5bfd2ad7e5b845ae9558ddc2f1a75063"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "d96badc1fdb74ad86d2f746508681e0a"
server_key_iv: "c7063a27e9f14973e60ec4e9472d9ca5"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.499233 4792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:30.500033 4812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.500249 4792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:30.500388 4792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "5bfd2ad7e5b845ae9558ddc2f1a75063"
format_stamp: "Formatted at 2026-08-10 15:20:30 on dist-test-slave-9zdj"
server_key: "d96badc1fdb74ad86d2f746508681e0a"
server_key_iv: "c7063a27e9f14973e60ec4e9472d9ca5"
server_key_version: "encryptionkey@0"
I20260810 15:20:30.500464 4792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:30.510694 4792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:30.512820 4792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:30.512995 4792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:30.513238 4792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:30.513639 4792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:30.513696 4792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.513738 4792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:30.513777 4792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:30.520078 4792 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:44471
I20260810 15:20:30.520133 4944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:44471 every 8 connection(s)
I20260810 15:20:30.520459 4792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:30.527683 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 4792
I20260810 15:20:30.527786 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:30.528039 4353 external_mini_cluster.cc:1428] Setting key f34187ebd79d60f247055e4f22423420
I20260810 15:20:30.530618 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.521361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:53043 (local address 127.4.64.126:37909)
0810 15:20:30.521434 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:30.521439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:30.521665 (+ 226us) server_negotiation.cc:407] Connection header received
0810 15:20:30.522270 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:30.522271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:30.522285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:30.522345 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:30.523353 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.523874 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.524908 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.525154 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.525253 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.527396 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:30.527397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:30.527397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:30.527404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:30.529255 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.529714 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.529715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.529715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.529767 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.529964 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.529964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.529965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.530061 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:30.530105 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:30.530418 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 15:20:30.530521 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:20:30.530680 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.521344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37909 (local address 127.4.64.67:53043)
0810 15:20:30.521520 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:30.521559 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:30.522148 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:30.522421 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:30.522425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:30.523031 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:30.523231 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:30.523237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.523988 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.523989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:30.524815 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:30.524817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.525262 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.525263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:30.525387 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.525905 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:30.525914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:30.527308 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:30.529341 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:30.529349 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:30.529352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:30.529604 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:30.529822 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:30.529823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:30.529823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:30.529923 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:30.530109 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:30.530114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:30.530229 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 15:20:30.530307 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 15:20:30.530429 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260810 15:20:30.531142 4945 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37909
I20260810 15:20:30.531247 4945 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:30.531445 4945 heartbeater.cc:511] Master 127.4.64.126:37909 requested a full tablet report, sending...
I20260810 15:20:30.531845 4414 ts_manager.cc:194] Registered new tserver with Master: 5bfd2ad7e5b845ae9558ddc2f1a75063 (127.4.64.67:44471)
I20260810 15:20:30.532222 4414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:53043
I20260810 15:20:30.540390 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:30.568672 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.559513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46206 (local address 127.4.64.65:38667)
0810 15:20:30.559728 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:30.559733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:30.559745 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:30.559774 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:30.559776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:30.559790 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:30.559872 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:30.560596 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.561453 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.562459 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.562836 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.562919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.565708 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:30.565718 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:30.565719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:30.565726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:30.567399 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.567810 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.567814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.567814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.567882 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.568103 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.568104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.568104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.568264 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:30.568393 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:30.568463 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 15:20:30.568536 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":123,"threads_started":1}
W20260810 15:20:30.569162 4538 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46206
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:30.596321 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:30.588614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.4.64.65:38667)
0810 15:20:30.588720 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:30.588722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:30.588733 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:30.588874 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:30.588876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:30.588892 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:30.588952 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:30.589578 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.590225 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.590961 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:30.591197 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:30.591259 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:30.593702 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:30.593703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:30.593704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:30.593711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:30.595026 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.595457 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.595458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.595458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.595518 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:30.595708 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:30.595709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:30.595709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:30.595812 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:30.595928 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:30.596083 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:20:30.596172 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 15:20:30.596940 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 4488
I20260810 15:20:30.606223 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 4641
I20260810 15:20:30.615270 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 4792
I20260810 15:20:30.621398 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 4376
2026-08-10T15:20:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2121 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4959](info): set up 2 sockets
Aug 10 15:20:30 dist-test-slave-9zdj krb5kdc[4959](info): commencing operation
krb5kdc: starting...
W20260810 15:20:31.748562 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:31 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:31.885910 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37671
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:31.971956 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:31.972169 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:31.972199 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:31.973403 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:31.973452 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:31.973466 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:31.973479 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:31.973491 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:31.975037 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:31.975306 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:31.975575 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:31.981127 4979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:31.981148 4980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:31.981168 4973 server_base.cc:1034] running on GCE node
W20260810 15:20:31.981148 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:31.981827 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:31.982113 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:31.983286 4973 hybrid_clock.cc:648] HybridClock initialized: now 1786375231983256 us; error 42 us; skew 500 ppm
Aug 10 15:20:31 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375231, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:31.986353 4973 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:31.986774 4973 webserver.cc:460] Webserver started at http://127.4.64.126:45713/ using document root <none> and password file <none>
I20260810 15:20:31.986990 4973 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:31.987039 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:31.987134 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:31.988276 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "1854129f74ce4482bfbd37dd2c79dd46"
format_stamp: "Formatted at 2026-08-10 15:20:31 on dist-test-slave-9zdj"
server_key: "9d034a7297e0670004f1f9059ce62943"
server_key_iv: "0aa2bd902c18e27d6b5afcda9c3ee1ba"
server_key_version: "encryptionkey@0"
I20260810 15:20:31.988597 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "1854129f74ce4482bfbd37dd2c79dd46"
format_stamp: "Formatted at 2026-08-10 15:20:31 on dist-test-slave-9zdj"
server_key: "9d034a7297e0670004f1f9059ce62943"
server_key_iv: "0aa2bd902c18e27d6b5afcda9c3ee1ba"
server_key_version: "encryptionkey@0"
I20260810 15:20:31.990021 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:31.990849 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:31.991062 4973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:31.991153 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "1854129f74ce4482bfbd37dd2c79dd46"
format_stamp: "Formatted at 2026-08-10 15:20:31 on dist-test-slave-9zdj"
server_key: "9d034a7297e0670004f1f9059ce62943"
server_key_iv: "0aa2bd902c18e27d6b5afcda9c3ee1ba"
server_key_version: "encryptionkey@0"
I20260810 15:20:31.991247 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:32.002099 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:32.004438 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:32.004577 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:32.009200 4973 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:37671
I20260810 15:20:32.009333 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:37671 every 8 connection(s)
I20260810 15:20:32.009755 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:32.010754 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:32.012087 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 4973
I20260810 15:20:32.012187 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:32.012342 4353 external_mini_cluster.cc:1428] Setting key b7296058bdca4d2a2edbd32fb6cc0369
I20260810 15:20:32.013670 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46: Bootstrap starting.
I20260810 15:20:32.014312 5045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:32.014616 5045 log.cc:824] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:32.015362 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46: No bootstrap required, opened a new log
I20260810 15:20:32.017412 5045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1854129f74ce4482bfbd37dd2c79dd46" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37671 } }
I20260810 15:20:32.017580 5045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:32.017650 5045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1854129f74ce4482bfbd37dd2c79dd46, State: Initialized, Role: FOLLOWER
I20260810 15:20:32.017810 5045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1854129f74ce4482bfbd37dd2c79dd46" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37671 } }
I20260810 15:20:32.017913 5045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:32.017958 5045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:32.018007 5045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:32.018579 5045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1854129f74ce4482bfbd37dd2c79dd46" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37671 } }
I20260810 15:20:32.018702 5045 leader_election.cc:302] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1854129f74ce4482bfbd37dd2c79dd46; no voters:
I20260810 15:20:32.018918 5045 leader_election.cc:288] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:32.018949 5050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:32.019066 5050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [term 1 LEADER]: Becoming Leader. State: Replica: 1854129f74ce4482bfbd37dd2c79dd46, State: Running, Role: LEADER
I20260810 15:20:32.019155 5050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1854129f74ce4482bfbd37dd2c79dd46" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37671 } }
I20260810 15:20:32.019198 5045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:32.019491 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1854129f74ce4482bfbd37dd2c79dd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1854129f74ce4482bfbd37dd2c79dd46" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37671 } } }
I20260810 15:20:32.019651 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:32.019874 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1854129f74ce4482bfbd37dd2c79dd46. Latest consensus state: current_term: 1 leader_uuid: "1854129f74ce4482bfbd37dd2c79dd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1854129f74ce4482bfbd37dd2c79dd46" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37671 } } }
I20260810 15:20:32.019948 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:32.020697 5057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:32.021679 5057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:32.023483 5057 catalog_manager.cc:1347] Generated new cluster ID: e58f4eed4f1d4754819096c09b9aac6f
I20260810 15:20:32.023535 5057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:32.024292 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.012960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.4.64.126:37671)
0810 15:20:32.013135 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.013145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.013181 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:20:32.013589 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.013595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.013698 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.013875 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.014561 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.015636 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.016410 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.016843 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.016978 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.019920 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.019924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.019925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.019938 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.022467 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.022911 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.022913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.022913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.023011 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.023277 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.023278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.023279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.023627 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.023782 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.023947 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.024088 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260810 15:20:32.029446 5057 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:32.030210 5057 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:32.037608 5057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1854129f74ce4482bfbd37dd2c79dd46: Generated new TSK 0
I20260810 15:20:32.037861 5057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:32.068413 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37671
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:32.156266 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:32.156525 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:32.156569 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:32.157807 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:32.157881 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:32.157924 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:32.159503 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:32.159806 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:32.160059 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:32.165724 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:32.165776 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:32.165738 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:32.166116 5071 server_base.cc:1034] running on GCE node
I20260810 15:20:32.166337 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:32.166548 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:32.167693 5071 hybrid_clock.cc:648] HybridClock initialized: now 1786375232167669 us; error 50 us; skew 500 ppm
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:32.175829 5071 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:32.176326 5071 webserver.cc:460] Webserver started at http://127.4.64.65:33143/ using document root <none> and password file <none>
I20260810 15:20:32.176594 5071 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:32.176671 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:32.176841 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:32.178081 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "678f361c9a554e36b81fcd1eca0d2069"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "3bf15ee54dcb64ae953600739a911677"
server_key_iv: "2aa63885556c9ba92d0c3353c8d1fde2"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.178503 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "678f361c9a554e36b81fcd1eca0d2069"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "3bf15ee54dcb64ae953600739a911677"
server_key_iv: "2aa63885556c9ba92d0c3353c8d1fde2"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.179987 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:32.180868 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.181066 5071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:32.181180 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "678f361c9a554e36b81fcd1eca0d2069"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "3bf15ee54dcb64ae953600739a911677"
server_key_iv: "2aa63885556c9ba92d0c3353c8d1fde2"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.181260 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:32.202551 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:32.205049 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:32.205262 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:32.205539 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:32.205948 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:32.206012 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.206046 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:32.206107 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.212447 5071 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:44125
I20260810 15:20:32.212497 5223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:44125 every 8 connection(s)
I20260810 15:20:32.212865 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:32.215209 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 5071
I20260810 15:20:32.215337 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:32.215524 4353 external_mini_cluster.cc:1428] Setting key 11db74cf67e14e84bf1c2a59b0bb3c5d
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:32.224673 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.213731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:35245 (local address 127.4.64.126:37671)
0810 15:20:32.213798 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.213800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.214024 (+ 224us) server_negotiation.cc:407] Connection header received
0810 15:20:32.215176 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.215177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.215236 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.215340 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.216130 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.216668 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.217990 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.218371 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.218426 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.221395 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.221397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.221397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.221407 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.222922 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.223460 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.223461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.223462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.223521 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.223893 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.223894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.223894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.224033 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.224113 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.224500 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.224592 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:20:32.224993 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.213692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37671 (local address 127.4.64.65:35245)
0810 15:20:32.213867 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.213940 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.215057 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.215438 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.215443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.215811 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:32.216020 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.216025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.216774 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.216775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.217869 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.217872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.218531 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.218532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.218658 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.219255 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:32.219265 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:32.221281 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:32.223012 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:32.223020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:32.223023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:32.223374 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:32.223591 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:32.223592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:32.223592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:32.223797 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:32.224125 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:32.224132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:32.224331 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.224468 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.224690 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260810 15:20:32.225785 5224 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37671
I20260810 15:20:32.225895 5224 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:32.226111 5224 heartbeater.cc:511] Master 127.4.64.126:37671 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
I20260810 15:20:32.226732 5007 ts_manager.cc:194] Registered new tserver with Master: 678f361c9a554e36b81fcd1eca0d2069 (127.4.64.65:44125)
I20260810 15:20:32.227514 5007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:35245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:32.245929 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37671
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:32.333415 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:32.333681 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:32.333709 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:32.334969 5229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:32.335024 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:32.335083 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:32.336686 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:32.336978 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:32.337231 5229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:32.342986 5235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:32.342983 5234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:32.343259 5229 server_base.cc:1034] running on GCE node
W20260810 15:20:32.342983 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:32.343647 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:32.343941 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:32.345124 5229 hybrid_clock.cc:648] HybridClock initialized: now 1786375232345110 us; error 56 us; skew 500 ppm
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:32.348054 5229 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:32.348580 5229 webserver.cc:460] Webserver started at http://127.4.64.66:33687/ using document root <none> and password file <none>
I20260810 15:20:32.348838 5229 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:32.348913 5229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:32.349076 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:32.350311 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "773520a6af514ac2b3b24e948f984613"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "21e17cbdb71a84a640e8d4c62807b653"
server_key_iv: "a4959403ee041207b23a643f2d7798d2"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.350703 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "773520a6af514ac2b3b24e948f984613"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "21e17cbdb71a84a640e8d4c62807b653"
server_key_iv: "a4959403ee041207b23a643f2d7798d2"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.352078 5229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:20:32.352850 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.353039 5229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:32.353152 5229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "773520a6af514ac2b3b24e948f984613"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "21e17cbdb71a84a640e8d4c62807b653"
server_key_iv: "a4959403ee041207b23a643f2d7798d2"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.353231 5229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:32.363153 5229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:32.365759 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:32.365945 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:32.366202 5229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:32.366559 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:32.366624 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.366668 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:32.366725 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.373103 5229 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:44757
I20260810 15:20:32.373154 5376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:44757 every 8 connection(s)
I20260810 15:20:32.373497 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:32.382539 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 5229
I20260810 15:20:32.382651 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:32.382840 4353 external_mini_cluster.cc:1428] Setting key 0bcb56979d30ae8c6ac2feec022d9c79
I20260810 15:20:32.384310 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.374457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:59555 (local address 127.4.64.126:37671)
0810 15:20:32.374539 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.374541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.374770 (+ 229us) server_negotiation.cc:407] Connection header received
0810 15:20:32.375843 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.375845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.375855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.375923 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.376642 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.377155 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.378438 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.378633 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.378692 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.380963 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.380964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.380964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.380973 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.382706 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.383187 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.383189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.383190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.383262 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.383560 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.383561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.383561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.383696 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.383793 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.384075 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.384170 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:20:32.384377 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.374443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37671 (local address 127.4.64.66:59555)
0810 15:20:32.374614 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.374658 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.375733 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.376030 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.376034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.376321 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:32.376524 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.376528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.377270 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.377271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.378335 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.378338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.378719 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.378719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.378823 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.379331 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:32.379340 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:32.380852 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:32.382805 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:32.382813 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:32.382815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:32.383117 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:32.383331 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:32.383332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:32.383332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:32.383488 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:32.383784 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:32.383793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:32.383933 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.384023 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.384167 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260810 15:20:32.384879 5378 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37671
I20260810 15:20:32.385000 5378 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:32.385215 5378 heartbeater.cc:511] Master 127.4.64.126:37671 requested a full tablet report, sending...
I20260810 15:20:32.385686 5007 ts_manager.cc:194] Registered new tserver with Master: 773520a6af514ac2b3b24e948f984613 (127.4.64.66:44757)
I20260810 15:20:32.386086 5007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:59555
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:32.421507 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37671
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:32.511960 5383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:32.512214 5383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:32.512243 5383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:32.513456 5383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:32.513505 5383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:32.513535 5383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:32.515246 5383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:32.515472 5383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:32.515681 5383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:32.521369 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:32.521369 5391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:32.521420 5389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:32.521680 5383 server_base.cc:1034] running on GCE node
I20260810 15:20:32.522050 5383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:32.522289 5383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:32.523473 5383 hybrid_clock.cc:648] HybridClock initialized: now 1786375232523451 us; error 44 us; skew 500 ppm
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:32.526494 5383 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:32.527041 5383 webserver.cc:460] Webserver started at http://127.4.64.67:36631/ using document root <none> and password file <none>
I20260810 15:20:32.527292 5383 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:32.527372 5383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:32.527532 5383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:32.528669 5383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "d5c055ea4004495eb992aa43f441b762"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "a5d33a5b1dd69fbc62af148236524dc1"
server_key_iv: "04a3ad3ca4375af4923930b534dd3cbb"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.529071 5383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "d5c055ea4004495eb992aa43f441b762"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "a5d33a5b1dd69fbc62af148236524dc1"
server_key_iv: "04a3ad3ca4375af4923930b534dd3cbb"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.530848 5383 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 15:20:32.531773 5399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.532016 5383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:32.532133 5383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "d5c055ea4004495eb992aa43f441b762"
format_stamp: "Formatted at 2026-08-10 15:20:32 on dist-test-slave-9zdj"
server_key: "a5d33a5b1dd69fbc62af148236524dc1"
server_key_iv: "04a3ad3ca4375af4923930b534dd3cbb"
server_key_version: "encryptionkey@0"
I20260810 15:20:32.532218 5383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:32.546483 5383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:32.549114 5383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:32.549304 5383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:32.549585 5383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:32.549962 5383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:32.550022 5383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.550066 5383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:32.550096 5383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:32.556397 5383 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:37305
I20260810 15:20:32.556481 5527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:37305 every 8 connection(s)
I20260810 15:20:32.556807 5383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:32.558207 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 5383
I20260810 15:20:32.558285 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:32.558425 4353 external_mini_cluster.cc:1428] Setting key 8ff9107137fcb59648853ea81c7867eb
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:32.567505 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.557663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:40437 (local address 127.4.64.126:37671)
0810 15:20:32.557723 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.557725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.557951 (+ 226us) server_negotiation.cc:407] Connection header received
0810 15:20:32.559223 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.559224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.559237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.559292 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.560073 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.560596 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.561724 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.561928 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.562015 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.564425 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.564427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.564428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.564436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.566012 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.566478 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.566479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.566480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.566545 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.566809 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.566810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.566811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.566939 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.567012 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.567320 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.567407 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:20:32.567755 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.557609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37671 (local address 127.4.64.67:40437)
0810 15:20:32.557818 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.557858 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.559099 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.559369 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.559373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.559717 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:32.559968 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.559972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.560696 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.560697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.561563 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.561566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.562028 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.562029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.562171 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.562783 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:32.562792 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:32.564318 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:32.566125 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:32.566134 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:32.566136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:32.566393 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:32.566614 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:32.566615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:32.566615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:32.566736 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:32.567014 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:32.567022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:32.567171 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.567286 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.567467 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260810 15:20:32.568341 5529 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37671
I20260810 15:20:32.568562 5529 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:32.568809 5529 heartbeater.cc:511] Master 127.4.64.126:37671 requested a full tablet report, sending...
I20260810 15:20:32.569262 5007 ts_manager.cc:194] Registered new tserver with Master: d5c055ea4004495eb992aa43f441b762 (127.4.64.67:37305)
I20260810 15:20:32.569734 5007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:40437
I20260810 15:20:32.570547 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:20:32.578414 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.572266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.4.64.126:37671)
0810 15:20:32.572304 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.572306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.572484 (+ 178us) server_negotiation.cc:407] Connection header received
0810 15:20:32.572508 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.572509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.572518 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.572572 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.573099 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.573586 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.574403 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.574586 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.574645 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.575841 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.575842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.575842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.575856 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.577190 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.577546 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.577547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.577547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.577603 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.577839 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.577840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.577840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.577992 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.578070 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.578205 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.578272 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:20:32.580854 5007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:32.582554 5007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:32.589854 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.585800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.4.64.65:44125)
0810 15:20:32.585985 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.585990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.586002 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:32.586055 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.586057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.586072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.586172 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:32.586345 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.587173 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.588574 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.589356 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.589441 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.589507 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.589699 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.589746 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260810 15:20:32.589884 5543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.585345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37305 (local address 127.0.0.1:49626)
0810 15:20:32.585466 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.585507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.585596 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.586009 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.586012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.586015 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:32.586099 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.586101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.587650 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.587652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.588580 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.588583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.589466 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.589467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.589557 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.589561 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.589658 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.589746 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20260810 15:20:32.590103 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.585959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.4.64.66:44757)
0810 15:20:32.586150 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.586155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.586163 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:32.586203 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.586205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.586219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.586312 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:32.586646 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.587700 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.588871 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.589375 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.589464 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.589525 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.589945 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.590000 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260810 15:20:32.590304 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.585468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.4.64.67:37305)
0810 15:20:32.585833 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.585839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.585851 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:32.585897 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.585900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.585925 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.586003 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:32.586351 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.587523 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.588676 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.589356 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.590075 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.590129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.590173 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.590222 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":114,"threads_started":1}
I20260810 15:20:32.590461 5546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.585714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:44757 (local address 127.0.0.1:51472)
0810 15:20:32.585822 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.585832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.585873 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.586413 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.586414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.586416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:32.586461 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.586462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.587798 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.587799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.588784 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.588786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.589556 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.589557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.589624 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.589654 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.589699 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.589770 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":10,"thread_start_us":76,"threads_started":1}
I20260810 15:20:32.590519 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.585466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44125 (local address 127.0.0.1:34886)
0810 15:20:32.585658 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.585668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.585711 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.586167 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.586168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.586170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:32.586219 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.586220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.587290 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.587291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.588491 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.588494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.589451 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.589452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.589539 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.589555 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.589655 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.589769 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":53,"thread_start_us":112,"threads_started":1}
I20260810 15:20:32.591113 5454 tablet_service.cc:1467] Processing CreateTablet for tablet a5ac8184e21745ecac5b653e9641849e (DEFAULT_TABLE table=test-table [id=51ba80dfe013448c8ddfb32c71eb66eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:32.591228 5304 tablet_service.cc:1467] Processing CreateTablet for tablet a5ac8184e21745ecac5b653e9641849e (DEFAULT_TABLE table=test-table [id=51ba80dfe013448c8ddfb32c71eb66eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:32.591614 5454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ac8184e21745ecac5b653e9641849e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:32.591614 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ac8184e21745ecac5b653e9641849e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:32.592583 5145 tablet_service.cc:1467] Processing CreateTablet for tablet a5ac8184e21745ecac5b653e9641849e (DEFAULT_TABLE table=test-table [id=51ba80dfe013448c8ddfb32c71eb66eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:32.593034 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5ac8184e21745ecac5b653e9641849e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:32.594871 5550 tablet_bootstrap.cc:492] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: Bootstrap starting.
I20260810 15:20:32.595376 5554 tablet_bootstrap.cc:492] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: Bootstrap starting.
I20260810 15:20:32.595772 5550 tablet_bootstrap.cc:654] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:32.596134 5554 tablet_bootstrap.cc:654] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:32.596441 5550 log.cc:824] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:32.596469 5555 tablet_bootstrap.cc:492] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: Bootstrap starting.
I20260810 15:20:32.596716 5554 log.cc:824] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:32.597100 5555 tablet_bootstrap.cc:654] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:32.597390 5550 tablet_bootstrap.cc:492] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: No bootstrap required, opened a new log
I20260810 15:20:32.597452 5550 ts_tablet_manager.cc:1397] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:32.597730 5555 log.cc:824] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:32.598204 5554 tablet_bootstrap.cc:492] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: No bootstrap required, opened a new log
I20260810 15:20:32.598294 5554 ts_tablet_manager.cc:1397] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:32.598615 5555 tablet_bootstrap.cc:492] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: No bootstrap required, opened a new log
I20260810 15:20:32.598686 5555 ts_tablet_manager.cc:1397] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:20:32.599370 5550 raft_consensus.cc:348] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.599607 5550 raft_consensus.cc:374] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:32.599722 5550 raft_consensus.cc:729] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773520a6af514ac2b3b24e948f984613, State: Initialized, Role: FOLLOWER
I20260810 15:20:32.599884 5550 consensus_queue.cc:260] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.600248 5378 heartbeater.cc:503] Master 127.4.64.126:37671 was elected leader, sending a full tablet report...
I20260810 15:20:32.600258 5550 ts_tablet_manager.cc:1428] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:32.600380 5554 raft_consensus.cc:348] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.600605 5554 raft_consensus.cc:374] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:32.600667 5554 raft_consensus.cc:729] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c055ea4004495eb992aa43f441b762, State: Initialized, Role: FOLLOWER
I20260810 15:20:32.600819 5554 consensus_queue.cc:260] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.600715 5555 raft_consensus.cc:348] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.600914 5555 raft_consensus.cc:374] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:32.600955 5555 raft_consensus.cc:729] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678f361c9a554e36b81fcd1eca0d2069, State: Initialized, Role: FOLLOWER
I20260810 15:20:32.601115 5554 ts_tablet_manager.cc:1428] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:32.601156 5555 consensus_queue.cc:260] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.601305 5529 heartbeater.cc:503] Master 127.4.64.126:37671 was elected leader, sending a full tablet report...
I20260810 15:20:32.601462 5555 ts_tablet_manager.cc:1428] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:32.601470 5224 heartbeater.cc:503] Master 127.4.64.126:37671 was elected leader, sending a full tablet report...
W20260810 15:20:32.624254 5379 tablet.cc:2369] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:20:32.713757 5225 tablet.cc:2369] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:32.727790 5559 raft_consensus.cc:482] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:32.727937 5559 raft_consensus.cc:504] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.728317 5559 leader_election.cc:288] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5c055ea4004495eb992aa43f441b762 (127.4.64.67:37305), 678f361c9a554e36b81fcd1eca0d2069 (127.4.64.65:44125)
I20260810 15:20:32.731668 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.728526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44125 (local address 127.4.64.66:37327)
0810 15:20:32.728574 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.728585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.728657 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.728848 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.728849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.728851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:32.728976 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.728979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.729948 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.729949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.730752 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.730755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.731356 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.731357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.731446 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.731450 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.731539 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.731589 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 15:20:32.731668 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.728619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:37327 (local address 127.4.64.65:44125)
0810 15:20:32.728695 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.728697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.728708 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:32.728738 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.728739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.728751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.728817 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:32.729200 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.729849 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.730875 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.731255 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.731358 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.731369 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.731553 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.731595 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 15:20:32.731828 5544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.728619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:51401 (local address 127.4.64.67:37305)
0810 15:20:32.728696 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.728698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.729062 (+ 364us) server_negotiation.cc:407] Connection header received
0810 15:20:32.729096 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.729097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.729109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.729154 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:32.729437 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.730066 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.731040 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.731450 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.731540 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.731550 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.731724 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.731765 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 15:20:32.731838 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.728526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37305 (local address 127.4.64.66:51401)
0810 15:20:32.728968 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:32.728979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:32.729049 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:32.729274 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:32.729275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:32.729276 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:32.729340 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.729341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.730177 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.730178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.730945 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:32.730947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.731544 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.731545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:32.731624 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.731627 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:32.731716 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:20:32.731754 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":185,"thread_start_us":154,"threads_started":1}
I20260810 15:20:32.732120 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ac8184e21745ecac5b653e9641849e" candidate_uuid: "773520a6af514ac2b3b24e948f984613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678f361c9a554e36b81fcd1eca0d2069" is_pre_election: true
I20260810 15:20:32.732337 5171 raft_consensus.cc:2454] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773520a6af514ac2b3b24e948f984613 in term 0.
I20260810 15:20:32.732394 5476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ac8184e21745ecac5b653e9641849e" candidate_uuid: "773520a6af514ac2b3b24e948f984613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c055ea4004495eb992aa43f441b762" is_pre_election: true
I20260810 15:20:32.732577 5476 raft_consensus.cc:2454] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773520a6af514ac2b3b24e948f984613 in term 0.
I20260810 15:20:32.732651 5248 leader_election.cc:302] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678f361c9a554e36b81fcd1eca0d2069, 773520a6af514ac2b3b24e948f984613; no voters:
I20260810 15:20:32.732807 5559 raft_consensus.cc:2781] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:32.732878 5559 raft_consensus.cc:482] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:32.732918 5559 raft_consensus.cc:3037] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:32.733743 5559 raft_consensus.cc:504] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.733901 5559 leader_election.cc:288] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [CANDIDATE]: Term 1 election: Requested vote from peers d5c055ea4004495eb992aa43f441b762 (127.4.64.67:37305), 678f361c9a554e36b81fcd1eca0d2069 (127.4.64.65:44125)
I20260810 15:20:32.734092 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ac8184e21745ecac5b653e9641849e" candidate_uuid: "773520a6af514ac2b3b24e948f984613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678f361c9a554e36b81fcd1eca0d2069"
I20260810 15:20:32.734143 5476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5ac8184e21745ecac5b653e9641849e" candidate_uuid: "773520a6af514ac2b3b24e948f984613" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c055ea4004495eb992aa43f441b762"
I20260810 15:20:32.734184 5171 raft_consensus.cc:3037] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:32.734234 5476 raft_consensus.cc:3037] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:32.735070 5476 raft_consensus.cc:2454] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773520a6af514ac2b3b24e948f984613 in term 1.
I20260810 15:20:32.735054 5171 raft_consensus.cc:2454] T a5ac8184e21745ecac5b653e9641849e P 678f361c9a554e36b81fcd1eca0d2069 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773520a6af514ac2b3b24e948f984613 in term 1.
I20260810 15:20:32.735245 5248 leader_election.cc:302] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678f361c9a554e36b81fcd1eca0d2069, 773520a6af514ac2b3b24e948f984613; no voters:
I20260810 15:20:32.735374 5559 raft_consensus.cc:2781] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:32.735510 5559 raft_consensus.cc:686] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [term 1 LEADER]: Becoming Leader. State: Replica: 773520a6af514ac2b3b24e948f984613, State: Running, Role: LEADER
I20260810 15:20:32.735617 5559 consensus_queue.cc:237] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } }
I20260810 15:20:32.736601 5006 catalog_manager.cc:5495] T a5ac8184e21745ecac5b653e9641849e P 773520a6af514ac2b3b24e948f984613 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773520a6af514ac2b3b24e948f984613 (127.4.64.66). New cstate: current_term: 1 leader_uuid: "773520a6af514ac2b3b24e948f984613" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5c055ea4004495eb992aa43f441b762" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "678f361c9a554e36b81fcd1eca0d2069" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773520a6af514ac2b3b24e948f984613" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 44757 } health_report { overall_health: HEALTHY } } }
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:32.778000 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.769570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.4.64.65:44125)
0810 15:20:32.769661 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.769663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.769710 (+ 47us) server_negotiation.cc:407] Connection header received
0810 15:20:32.769789 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.769790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.769801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.769861 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.770516 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.771169 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.771907 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.772111 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.772169 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.774749 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.774763 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.774763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.774771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.776588 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.776980 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.776984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.776985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.777099 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.777306 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.777307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.777307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.777476 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.777602 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.777748 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.777846 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:32.805230 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.796807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.4.64.65:44125)
0810 15:20:32.796861 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.796863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.796941 (+ 78us) server_negotiation.cc:407] Connection header received
0810 15:20:32.796992 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.796993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.797002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.797054 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.797938 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.798535 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.799509 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.799731 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.799805 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.802498 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.802500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.802500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.802508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.804051 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.804415 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.804416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.804416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.804472 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.804646 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.804646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.804647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.804749 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.804824 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.805072 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.805157 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 15:20:32.807392 5530 tablet.cc:2369] T a5ac8184e21745ecac5b653e9641849e P d5c055ea4004495eb992aa43f441b762: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[4959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:32.834359 5548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:32.824951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.4.64.65:44125)
0810 15:20:32.825092 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:32.825095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:32.825266 (+ 171us) server_negotiation.cc:407] Connection header received
0810 15:20:32.825326 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:32.825327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:32.825339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:32.825416 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:32.826124 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.826636 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.827471 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:32.827670 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:32.827761 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:32.831365 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:32.831367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:32.831367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:32.831376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:32.832948 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.833383 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.833384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.833385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.833500 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:32.833802 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:32.833803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:32.833804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:32.833927 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:32.834040 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:32.834153 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 15:20:32.834244 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 15:20:32.835320 5120 tablet_service.cc:2908] Rejecting scan request for tablet a5ac8184e21745ecac5b653e9641849e: Uninitialized: clean time has not yet been initialized
I20260810 15:20:32.836688 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 5071
I20260810 15:20:32.845541 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 5229
I20260810 15:20:32.855132 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 5383
I20260810 15:20:32.863579 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 4973
2026-08-10T15:20:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2242 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[5583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[5583](info): set up 2 sockets
Aug 10 15:20:32 dist-test-slave-9zdj krb5kdc[5583](info): commencing operation
krb5kdc: starting...
W20260810 15:20:33.999536 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:34.136787 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:36923
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:36923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:34.224188 5597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:34.224452 5597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:34.224493 5597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:34.225728 5597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:34.225790 5597 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 15:20:34.225811 5597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:34.225833 5597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:34.225862 5597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:34.225891 5597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:34.227556 5597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:36923
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:34.227859 5597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:34.228101 5597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:34.233217 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.233402 5597 server_base.cc:1034] running on GCE node
W20260810 15:20:34.233209 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:34.233209 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.233817 5597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:34.234186 5597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:34.235356 5597 hybrid_clock.cc:648] HybridClock initialized: now 1786375234235328 us; error 48 us; skew 500 ppm
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:34.238395 5597 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:34.238869 5597 webserver.cc:460] Webserver started at http://127.4.64.126:42439/ using document root <none> and password file <none>
I20260810 15:20:34.239125 5597 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:34.239199 5597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:34.239368 5597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:34.240469 5597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "cec9a435fbef4dc1bfac66e17d5939e8"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "0c87375cc05eba69999d69c611a4535c"
server_key_iv: "6db2f342fda22c3b7cce4981ae6b7c3d"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.240849 5597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "cec9a435fbef4dc1bfac66e17d5939e8"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "0c87375cc05eba69999d69c611a4535c"
server_key_iv: "6db2f342fda22c3b7cce4981ae6b7c3d"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.242446 5597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:34.243336 5614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.243549 5597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:34.243665 5597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "cec9a435fbef4dc1bfac66e17d5939e8"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "0c87375cc05eba69999d69c611a4535c"
server_key_iv: "6db2f342fda22c3b7cce4981ae6b7c3d"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.243747 5597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:34.267827 5597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:34.270301 5597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:34.270498 5597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:34.275873 5597 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:36923
I20260810 15:20:34.275977 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:36923 every 8 connection(s)
I20260810 15:20:34.276353 5597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:34.277154 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:34.279740 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: Bootstrap starting.
I20260810 15:20:34.280440 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:34.280766 5682 log.cc:824] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:34.281574 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: No bootstrap required, opened a new log
I20260810 15:20:34.283298 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 5597
I20260810 15:20:34.283481 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:34.283660 4353 external_mini_cluster.cc:1428] Setting key 26ad1d76ea749043b3b743ec3b8e7976
I20260810 15:20:34.284291 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36923 } }
I20260810 15:20:34.284528 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:34.284583 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec9a435fbef4dc1bfac66e17d5939e8, State: Initialized, Role: FOLLOWER
I20260810 15:20:34.284744 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36923 } }
I20260810 15:20:34.284857 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:34.284893 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:34.284977 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:34.285840 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36923 } }
I20260810 15:20:34.285936 5682 leader_election.cc:302] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec9a435fbef4dc1bfac66e17d5939e8; no voters:
I20260810 15:20:34.286127 5682 leader_election.cc:288] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:34.286185 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:34.286355 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [term 1 LEADER]: Becoming Leader. State: Replica: cec9a435fbef4dc1bfac66e17d5939e8, State: Running, Role: LEADER
I20260810 15:20:34.286429 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:34.286518 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36923 } }
I20260810 15:20:34.287281 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36923 } } }
I20260810 15:20:34.287370 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:34.287596 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec9a435fbef4dc1bfac66e17d5939e8. Latest consensus state: current_term: 1 leader_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9a435fbef4dc1bfac66e17d5939e8" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36923 } } }
I20260810 15:20:34.287750 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:34.288559 5706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:34.288733 5706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:34.288827 5699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:34.289014 5699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:34.291015 5699 catalog_manager.cc:1347] Generated new cluster ID: 39474ad1e00841949ef991c357ac0c6a
I20260810 15:20:34.291067 5699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:34.295419 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.284275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.4.64.126:36923)
0810 15:20:34.284481 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.284490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.284517 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:20:34.284670 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.284677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.284733 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.284876 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.285736 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.287382 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.288548 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.289191 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.289323 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.291979 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.291983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.291983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.291995 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.293825 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.294208 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.294212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.294212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.294323 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.294546 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.294547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.294547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.294808 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.294980 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.295064 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.295214 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260810 15:20:34.299773 5699 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:34.300482 5699 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:34.304011 5699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: Generated new TSK 0
I20260810 15:20:34.304185 5699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:34.336886 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36923
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:34.423063 5712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:34.423285 5712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:34.423314 5712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:34.424484 5712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:34.424526 5712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:34.424547 5712 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:20:34.424564 5712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:34.426048 5712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:34.426321 5712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:34.426539 5712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:34.431890 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:34.431874 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:34.431874 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.432891 5712 server_base.cc:1034] running on GCE node
I20260810 15:20:34.433125 5712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:34.433400 5712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:34.434585 5712 hybrid_clock.cc:648] HybridClock initialized: now 1786375234434526 us; error 80 us; skew 500 ppm
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:34.437372 5712 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:34.437899 5712 webserver.cc:460] Webserver started at http://127.4.64.65:45581/ using document root <none> and password file <none>
I20260810 15:20:34.438169 5712 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:34.438251 5712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:34.438406 5712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:34.439589 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "cb91403ccb8a4e9fac7cd30459df57f5"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "98e5c3311f86902c9edef6912b47ea4b"
server_key_iv: "6314b8dda21a248cd671267ed5b98b85"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.439971 5712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "cb91403ccb8a4e9fac7cd30459df57f5"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "98e5c3311f86902c9edef6912b47ea4b"
server_key_iv: "6314b8dda21a248cd671267ed5b98b85"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.441447 5712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:34.442365 5728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.442554 5712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:34.442651 5712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "cb91403ccb8a4e9fac7cd30459df57f5"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "98e5c3311f86902c9edef6912b47ea4b"
server_key_iv: "6314b8dda21a248cd671267ed5b98b85"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.442739 5712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:34.453066 5712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:34.455454 5712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:34.455636 5712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:34.456187 5712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:34.456249 5712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.456284 5712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:34.456305 5712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:34.463958 5712 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:43239
I20260810 15:20:34.464035 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:43239 every 8 connection(s)
I20260810 15:20:34.464354 5712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:34.466948 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.456874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:41393 (local address 127.4.64.126:36923)
0810 15:20:34.456954 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.456957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.457145 (+ 188us) server_negotiation.cc:407] Connection header received
0810 15:20:34.457651 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.457653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.457666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.457766 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.458616 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.459472 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.460607 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.460965 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.461041 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.464096 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.464097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.464098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.464106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.465752 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.466176 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.466177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.466177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.466233 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.466424 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.466425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.466425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.466530 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.466580 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.466800 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.466874 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:20:34.467090 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.456796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.4.64.65:41393)
0810 15:20:34.457009 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.457050 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.457536 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.457830 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.457833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.458197 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:34.458485 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.458494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.459579 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.459580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.460503 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.460506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.461083 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.461085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.461269 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.461915 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:34.461926 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:34.463980 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:34.465869 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.465877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.465879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.466105 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.466275 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.466276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.466276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.466371 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.466572 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:34.466575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:34.466690 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.466768 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.466899 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260810 15:20:34.467687 5857 heartbeater.cc:347] Connected to a master server at 127.4.64.126:36923
I20260810 15:20:34.467808 5857 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:34.468014 5857 heartbeater.cc:511] Master 127.4.64.126:36923 requested a full tablet report, sending...
I20260810 15:20:34.468659 5631 ts_manager.cc:194] Registered new tserver with Master: cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239)
I20260810 15:20:34.469650 5631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:41393
I20260810 15:20:34.473752 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 5712
I20260810 15:20:34.473869 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:34.474073 4353 external_mini_cluster.cc:1428] Setting key b2cfe91b35acba06b4f4dcbb016dc061
I20260810 15:20:34.474721 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.468928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:51421 (local address 127.4.64.126:36923)
0810 15:20:34.468974 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.468976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.469061 (+ 85us) server_negotiation.cc:407] Connection header received
0810 15:20:34.469118 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.469119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.469127 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.469173 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.469534 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.470202 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.470984 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.471165 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.471232 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.472179 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.472180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.472180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.472186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.473443 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.473840 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.473841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.473842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.473909 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.474122 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.474123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.474124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.474278 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.474358 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.474563 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.474648 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:34.474767 5864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.468844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.4.64.65:51421)
0810 15:20:34.468987 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.468999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.469063 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.469178 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.469178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.469392 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:34.469466 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.469468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.470299 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.470300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.470908 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.470910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.471234 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.471234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.471290 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.471758 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:34.471763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:34.472100 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:34.473519 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.473520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.473521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.473761 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.473979 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.473980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.473980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.474052 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.474378 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:34.474379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:34.474446 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.474543 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.474653 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:34.506088 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36923
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:34.596100 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:34.596338 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:34.596365 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:34.597534 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:34.597573 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:34.597594 5868 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:20:34.597692 5868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:34.599232 5868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:34.599460 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:34.599666 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:34.605079 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:34.605178 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.605332 5868 server_base.cc:1034] running on GCE node
W20260810 15:20:34.605089 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.605758 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:34.606045 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:34.607277 5868 hybrid_clock.cc:648] HybridClock initialized: now 1786375234607316 us; error 83 us; skew 500 ppm
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:34.610000 5868 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:34.610415 5868 webserver.cc:460] Webserver started at http://127.4.64.66:33443/ using document root <none> and password file <none>
I20260810 15:20:34.610625 5868 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:34.610677 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:34.610776 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:34.611882 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "293080af495a4fe4a157d19944f07f57"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "af0d5b0294499a5df04fcb409c4f365f"
server_key_iv: "7d55f1d674572661a008ace6ac126bb1"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.612200 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "293080af495a4fe4a157d19944f07f57"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "af0d5b0294499a5df04fcb409c4f365f"
server_key_iv: "7d55f1d674572661a008ace6ac126bb1"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.613549 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:20:34.614387 5887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.614571 5868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:34.614670 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "293080af495a4fe4a157d19944f07f57"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "af0d5b0294499a5df04fcb409c4f365f"
server_key_iv: "7d55f1d674572661a008ace6ac126bb1"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.614748 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:34.632409 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:34.634521 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:34.634682 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:34.635442 5868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:34.635529 5868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.635584 5868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:34.635638 5868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.642709 5868 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:37093
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:34.642781 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:37093 every 8 connection(s)
I20260810 15:20:34.643148 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:34.646559 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.635858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:38475 (local address 127.4.64.126:36923)
0810 15:20:34.635920 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.635922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.636164 (+ 242us) server_negotiation.cc:407] Connection header received
0810 15:20:34.636734 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.636735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.636747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.636805 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.637743 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.638632 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.640347 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.640671 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.640756 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.643336 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.643338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.643338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.643348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.644869 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.645363 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.645365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.645365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.645433 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.645818 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.645819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.645820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.645950 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.646028 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.646371 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.646465 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:20:34.646723 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.635849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.4.64.66:38475)
0810 15:20:34.636025 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.636065 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.636614 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.636885 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.636889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.637245 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:34.637559 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.637565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.638775 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.638776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.640239 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.640243 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.640779 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.640780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.640896 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.641481 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:34.641492 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:34.643206 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:34.644978 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.644986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.644989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.645276 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.645512 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.645512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.645513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.645722 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.646030 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:34.646038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:34.646194 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.646306 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.646485 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260810 15:20:34.647267 6013 heartbeater.cc:347] Connected to a master server at 127.4.64.126:36923
I20260810 15:20:34.647410 6013 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:34.647625 6013 heartbeater.cc:511] Master 127.4.64.126:36923 requested a full tablet report, sending...
I20260810 15:20:34.648025 5632 ts_manager.cc:194] Registered new tserver with Master: 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093)
I20260810 15:20:34.648553 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:38475
I20260810 15:20:34.652587 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 5868
I20260810 15:20:34.652705 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:34.652901 4353 external_mini_cluster.cc:1428] Setting key 85277128be63b077da65e16ab6651c75
I20260810 15:20:34.654482 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.648453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:36769 (local address 127.4.64.126:36923)
0810 15:20:34.648690 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.648692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.648701 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:20:34.648721 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.648722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.648733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.648787 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.649256 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.649895 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.650584 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.650778 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.650836 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.651710 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.651711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.651711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.651717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.653236 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.653643 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.653644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.653645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.653732 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.653967 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.653968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.653968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.654110 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.654165 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.654353 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.654417 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260810 15:20:34.654628 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.648382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.4.64.66:36769)
0810 15:20:34.648512 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.648532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.648590 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.648807 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.648808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.649038 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:34.649122 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.649124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.649984 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.649985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.650501 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.650502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.650849 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.650850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.650907 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.651320 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:34.651324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:34.651627 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:34.653329 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.653330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.653330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.653540 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.653808 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.653809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.653810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.653895 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.654182 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:34.654183 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:34.654238 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.654324 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.654462 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:34.685316 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36923
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:34.772014 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:34.772287 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:34.772332 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:34.773703 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:34.773765 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:34.773797 6022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:20:34.773835 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:34.775345 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:34.775636 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:34.775887 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:34.781113 6030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:34.781103 6032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.781175 6022 server_base.cc:1034] running on GCE node
W20260810 15:20:34.781103 6028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:34.781525 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:34.781805 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:34.782953 6022 hybrid_clock.cc:648] HybridClock initialized: now 1786375234782933 us; error 37 us; skew 500 ppm
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:34.785821 6022 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:34.786273 6022 webserver.cc:460] Webserver started at http://127.4.64.67:43463/ using document root <none> and password file <none>
I20260810 15:20:34.786583 6022 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:34.786635 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:34.786820 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:34.788091 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "3a68484648e74b30b4d436ce5acc72bd"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "92af6328c33c926426a38de5db3f5a83"
server_key_iv: "44976e0cef96faeb5bb0b0a54f4f943c"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.788529 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "3a68484648e74b30b4d436ce5acc72bd"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "92af6328c33c926426a38de5db3f5a83"
server_key_iv: "44976e0cef96faeb5bb0b0a54f4f943c"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.790019 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:34.790889 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.791077 6022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:34.791195 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "3a68484648e74b30b4d436ce5acc72bd"
format_stamp: "Formatted at 2026-08-10 15:20:34 on dist-test-slave-9zdj"
server_key: "92af6328c33c926426a38de5db3f5a83"
server_key_iv: "44976e0cef96faeb5bb0b0a54f4f943c"
server_key_version: "encryptionkey@0"
I20260810 15:20:34.791285 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:34.802569 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:34.805295 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:34.805485 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:34.806083 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:34.806160 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.806217 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:34.806244 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:34.813498 6022 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43449
I20260810 15:20:34.813553 6173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43449 every 8 connection(s)
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:34.813977 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:34.817368 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.806678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:43039 (local address 127.4.64.126:36923)
0810 15:20:34.806725 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.806727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.806966 (+ 239us) server_negotiation.cc:407] Connection header received
0810 15:20:34.807471 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.807473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.807487 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.807570 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.808499 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.809361 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.811008 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.811370 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.811460 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.814429 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.814431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.814431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.814453 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.816027 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.816431 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.816431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.816432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.816497 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.816724 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.816725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.816725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.816868 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.816922 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.817175 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.817248 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:34.817463 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.806675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.4.64.67:43039)
0810 15:20:34.806853 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.806889 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.807364 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.807671 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.807676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.808065 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:34.808381 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.808388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.809481 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.809482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.810886 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.810889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.811476 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.811477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.811606 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.812207 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:34.812216 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:34.814299 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:34.816113 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.816120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.816122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.816357 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.816567 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.816567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.816567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.816662 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.816927 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:34.816930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:34.817055 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.817144 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.817296 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260810 15:20:34.818063 6174 heartbeater.cc:347] Connected to a master server at 127.4.64.126:36923
I20260810 15:20:34.818161 6174 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:34.818488 6174 heartbeater.cc:511] Master 127.4.64.126:36923 requested a full tablet report, sending...
I20260810 15:20:34.818948 5631 ts_manager.cc:194] Registered new tserver with Master: 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67:43449)
I20260810 15:20:34.819404 5631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:43039
I20260810 15:20:34.822146 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 6022
I20260810 15:20:34.822254 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:34.822443 4353 external_mini_cluster.cc:1428] Setting key b8854902e916b84e0c89a7cff11570a9
I20260810 15:20:34.823681 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:20:34.825482 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.819310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:46073 (local address 127.4.64.126:36923)
0810 15:20:34.819353 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.819355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.819543 (+ 188us) server_negotiation.cc:407] Connection header received
0810 15:20:34.819570 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.819571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.819583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.819645 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.819978 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.820481 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.821195 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.821409 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.821494 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.822675 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.822677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.822677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.822685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.824329 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.824738 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.824739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.824739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.824796 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.825000 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.825001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.825001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.825107 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.825171 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.825351 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.825418 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:20:34.825639 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.819258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.4.64.67:46073)
0810 15:20:34.819424 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.819445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.819517 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.819648 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.819648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.819831 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:34.819892 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.819894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.820583 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.820585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.821106 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.821108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.821497 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.821498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.821577 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.822146 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:34.822152 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:34.822566 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:34.824423 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.824424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.824424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.824658 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.824862 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:34.824863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:34.824864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:34.824939 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:34.825184 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:34.825185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:34.825245 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.825319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.825447 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:20:34 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:34.849706 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.841999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.4.64.126:36923)
0810 15:20:34.842050 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.842052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.842081 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:20:34.842109 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.842110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.842119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.842164 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:34.842780 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.843283 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.844049 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.844248 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.844315 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.846946 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:34.846947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:34.846947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:34.846954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:34.848468 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.848842 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.848843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.848844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.848912 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:34.849120 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:34.849121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:34.849122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:34.849241 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:34.849355 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.849465 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.849552 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:34.851539 5631 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:34.853279 5631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:34.860189 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.856296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.4.64.66:37093)
0810 15:20:34.856535 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.856539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.856824 (+ 285us) server_negotiation.cc:407] Connection header received
0810 15:20:34.856872 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.856874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.856889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.856967 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:34.857095 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.858138 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.859236 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.859769 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.859878 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.859948 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.860054 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.860094 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":120,"threads_started":1}
I20260810 15:20:34.860191 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.856177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:37093 (local address 127.0.0.1:60820)
0810 15:20:34.856598 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.856607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.856768 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.856985 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.856986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.856988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:34.857026 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.857027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.858232 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.858233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.859143 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.859146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.859863 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.859865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.859948 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.859963 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.860040 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.860094 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":71,"threads_started":1}
I20260810 15:20:34.860463 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.856013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43449 (local address 127.0.0.1:46226)
0810 15:20:34.856301 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.856310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.856352 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.856622 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.856623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.856626 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:34.856686 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.856688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.857757 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.857758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.859186 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.859189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.860184 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.860186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.860282 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.860286 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.860336 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.860381 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":81,"thread_start_us":67,"threads_started":1}
I20260810 15:20:34.860570 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.855926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.4.64.65:43239)
0810 15:20:34.856129 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.856135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.856361 (+ 226us) server_negotiation.cc:407] Connection header received
0810 15:20:34.856409 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.856411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.856429 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.856502 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:34.856759 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.857885 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.859292 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.860018 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.860091 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.860145 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.860430 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.860468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260810 15:20:34.860606 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.855778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:43239 (local address 127.0.0.1:47226)
0810 15:20:34.856134 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.856189 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.856265 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.856583 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.856587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.856589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:34.856676 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.856678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.857972 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.857973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.859188 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.859189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.860186 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.860187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.860280 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.860284 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.860336 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.860380 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":42,"thread_start_us":76,"threads_started":1}
I20260810 15:20:34.860821 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.856136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.4.64.67:43449)
0810 15:20:34.856325 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.856329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.856346 (+ 17us) server_negotiation.cc:407] Connection header received
0810 15:20:34.856443 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.856445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.856463 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.856547 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:34.856771 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.857602 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.859292 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.860023 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.860091 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.860145 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.860680 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.860724 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260810 15:20:34.861321 5940 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa47807d98d45da9597ee68623c536a (DEFAULT_TABLE table=test-table [id=11303d41e06d48a6b3f780b1fffb9a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:34.861400 5786 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa47807d98d45da9597ee68623c536a (DEFAULT_TABLE table=test-table [id=11303d41e06d48a6b3f780b1fffb9a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:34.861755 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa47807d98d45da9597ee68623c536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:34.861755 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa47807d98d45da9597ee68623c536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:34.862138 6097 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa47807d98d45da9597ee68623c536a (DEFAULT_TABLE table=test-table [id=11303d41e06d48a6b3f780b1fffb9a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:34.862519 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa47807d98d45da9597ee68623c536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:34.865041 6201 tablet_bootstrap.cc:492] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Bootstrap starting.
I20260810 15:20:34.865787 6203 tablet_bootstrap.cc:492] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Bootstrap starting.
I20260810 15:20:34.866277 6201 tablet_bootstrap.cc:654] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:34.866377 6203 tablet_bootstrap.cc:654] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:34.866616 6201 log.cc:824] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:34.866721 6203 log.cc:824] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:34.867136 6202 tablet_bootstrap.cc:492] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Bootstrap starting.
I20260810 15:20:34.867441 6201 tablet_bootstrap.cc:492] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: No bootstrap required, opened a new log
I20260810 15:20:34.867499 6201 ts_tablet_manager.cc:1397] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:34.867722 6203 tablet_bootstrap.cc:492] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: No bootstrap required, opened a new log
I20260810 15:20:34.867805 6202 tablet_bootstrap.cc:654] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:34.867837 6203 ts_tablet_manager.cc:1397] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:20:34.868162 6202 log.cc:824] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:34.868992 6202 tablet_bootstrap.cc:492] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: No bootstrap required, opened a new log
I20260810 15:20:34.869059 6202 ts_tablet_manager.cc:1397] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:34.869570 6201 raft_consensus.cc:348] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.869809 6201 raft_consensus.cc:374] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:34.869859 6201 raft_consensus.cc:729] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a68484648e74b30b4d436ce5acc72bd, State: Initialized, Role: FOLLOWER
I20260810 15:20:34.869997 6201 consensus_queue.cc:260] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.870086 6203 raft_consensus.cc:348] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.870253 6203 raft_consensus.cc:374] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:34.870287 6201 ts_tablet_manager.cc:1428] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:34.870297 6203 raft_consensus.cc:729] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb91403ccb8a4e9fac7cd30459df57f5, State: Initialized, Role: FOLLOWER
I20260810 15:20:34.870450 6174 heartbeater.cc:503] Master 127.4.64.126:36923 was elected leader, sending a full tablet report...
I20260810 15:20:34.870425 6203 consensus_queue.cc:260] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.870685 6203 ts_tablet_manager.cc:1428] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:34.870713 5857 heartbeater.cc:503] Master 127.4.64.126:36923 was elected leader, sending a full tablet report...
I20260810 15:20:34.871162 6202 raft_consensus.cc:348] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.871340 6202 raft_consensus.cc:374] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:34.871392 6202 raft_consensus.cc:729] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293080af495a4fe4a157d19944f07f57, State: Initialized, Role: FOLLOWER
I20260810 15:20:34.871507 6202 consensus_queue.cc:260] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.871795 6202 ts_tablet_manager.cc:1428] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:34.871873 6013 heartbeater.cc:503] Master 127.4.64.126:36923 was elected leader, sending a full tablet report...
W20260810 15:20:34.893808 6014 tablet.cc:2369] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:20:34.965472 5859 tablet.cc:2369] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:34.971510 6207 raft_consensus.cc:482] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:34.971661 6207 raft_consensus.cc:504] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.971992 6207 leader_election.cc:288] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239), 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093)
I20260810 15:20:34.975221 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.972307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:35363 (local address 127.4.64.65:43239)
0810 15:20:34.972369 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.972371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.972386 (+ 15us) server_negotiation.cc:407] Connection header received
0810 15:20:34.972420 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.972421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.972435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.972529 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:34.972716 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.973217 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.974482 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.974851 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.974958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.974970 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.975133 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.975165 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:20:34.975263 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.972234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:43239 (local address 127.4.64.67:35363)
0810 15:20:34.972299 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.972312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.972362 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.972540 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.972541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.972543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:34.972635 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.972637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.973309 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.973310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.974374 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.974377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.974967 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.974969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.975055 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.975058 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.975121 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.975159 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 15:20:34.975576 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.972369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:37093 (local address 127.4.64.67:38167)
0810 15:20:34.972611 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:34.972621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:34.972699 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:34.972837 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:34.972838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:34.972839 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:34.972919 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.972922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.973733 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.973734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.974591 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:34.974593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.975304 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.975305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:34.975396 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.975401 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:20:34.975464 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:20:34.975500 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":159,"threads_started":1}
I20260810 15:20:34.975632 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:34.972528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:38167 (local address 127.4.64.66:37093)
0810 15:20:34.972577 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:34.972579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:34.972686 (+ 107us) server_negotiation.cc:407] Connection header received
0810 15:20:34.972755 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:34.972756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:34.972769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:34.972832 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:34.972992 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.973611 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.974687 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:34.975208 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:34.975274 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:34.975285 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:34.975490 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 15:20:34.975547 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:20:34.975791 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa47807d98d45da9597ee68623c536a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" is_pre_election: true
I20260810 15:20:34.976001 5806 raft_consensus.cc:2454] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 0.
I20260810 15:20:34.976001 5961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa47807d98d45da9597ee68623c536a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293080af495a4fe4a157d19944f07f57" is_pre_election: true
I20260810 15:20:34.976133 5961 raft_consensus.cc:2454] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 0.
I20260810 15:20:34.976253 6044 leader_election.cc:302] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a68484648e74b30b4d436ce5acc72bd, cb91403ccb8a4e9fac7cd30459df57f5; no voters:
I20260810 15:20:34.976434 6207 raft_consensus.cc:2781] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:34.976495 6207 raft_consensus.cc:482] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:34.976511 6207 raft_consensus.cc:3037] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:34.977331 6207 raft_consensus.cc:504] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.977502 6207 leader_election.cc:288] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 election: Requested vote from peers cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239), 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093)
I20260810 15:20:34.977687 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa47807d98d45da9597ee68623c536a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb91403ccb8a4e9fac7cd30459df57f5"
I20260810 15:20:34.977725 5961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa47807d98d45da9597ee68623c536a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293080af495a4fe4a157d19944f07f57"
I20260810 15:20:34.977802 5961 raft_consensus.cc:3037] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:34.977805 5806 raft_consensus.cc:3037] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:34.978482 5961 raft_consensus.cc:2454] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 1.
I20260810 15:20:34.978546 5806 raft_consensus.cc:2454] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 1.
I20260810 15:20:34.978647 6044 leader_election.cc:302] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293080af495a4fe4a157d19944f07f57, 3a68484648e74b30b4d436ce5acc72bd; no voters:
I20260810 15:20:34.978760 6207 raft_consensus.cc:2781] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:34.978942 6207 raft_consensus.cc:686] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 1 LEADER]: Becoming Leader. State: Replica: 3a68484648e74b30b4d436ce5acc72bd, State: Running, Role: LEADER
I20260810 15:20:34.979063 6207 consensus_queue.cc:237] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } }
I20260810 15:20:34.979995 5632 catalog_manager.cc:5495] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67). New cstate: current_term: 1 leader_uuid: "3a68484648e74b30b4d436ce5acc72bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } health_report { overall_health: UNKNOWN } } }
I20260810 15:20:35.008421 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.005739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.4.64.67:43449)
0810 15:20:35.005796 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.005799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.005809 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:35.005909 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.005910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.005938 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.006019 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:35.006196 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.006690 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.007530 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.007730 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.007810 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.008064 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.008165 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.008254 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.008298 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.008324 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:20:35.010988 5961 raft_consensus.cc:1261] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Refusing update from remote peer 3a68484648e74b30b4d436ce5acc72bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:35.011018 5806 raft_consensus.cc:1261] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Refusing update from remote peer 3a68484648e74b30b4d436ce5acc72bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:35.011431 6211 consensus_queue.cc:1035] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:35.011583 6207 consensus_queue.cc:1035] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:35.014489 6217 mvcc.cc:205] Tried to move back new op lower bound from 7316992962601799680 to 7316992962475139072. Current Snapshot: MvccSnapshot[applied={T|T < 7316992962601799680}]
I20260810 15:20:35.014876 6216 mvcc.cc:205] Tried to move back new op lower bound from 7316992962601799680 to 7316992962475139072. Current Snapshot: MvccSnapshot[applied={T|T < 7316992962601799680}]
I20260810 15:20:35.017838 5632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:35.017948 5632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:35.018815 5632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: Sending DeleteTablet for 3 replicas of tablet 9aa47807d98d45da9597ee68623c536a
I20260810 15:20:35.019235 5940 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa47807d98d45da9597ee68623c536a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:35 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:60820
I20260810 15:20:35.019299 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa47807d98d45da9597ee68623c536a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:35 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:46226
I20260810 15:20:35.019299 5786 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa47807d98d45da9597ee68623c536a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:35 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47226
I20260810 15:20:35.019629 5632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 15:20:35.019701 6228 tablet_replica.cc:330] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: stopping tablet replica
W20260810 15:20:35.019784 5632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:35.019840 6228 raft_consensus.cc:2229] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:35.019948 6228 raft_consensus.cc:2258] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:35.020433 6228 ts_tablet_manager.cc:1905] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:35.021062 6227 tablet_replica.cc:330] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: stopping tablet replica
I20260810 15:20:35.021071 5786 tablet_service.cc:1467] Processing CreateTablet for tablet f11d1d0fca224e07abb5e1ee6fa1519a (DEFAULT_TABLE table=test-table [id=5f2592329abb47cb8f32fc3b3f17c1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:35.021190 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11d1d0fca224e07abb5e1ee6fa1519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:35.021311 6227 raft_consensus.cc:2229] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:35.021438 6227 raft_consensus.cc:2258] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:35.021924 6227 ts_tablet_manager.cc:1905] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:35.022449 6228 ts_tablet_manager.cc:1918] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:35.022524 6228 log.cc:1196] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/9aa47807d98d45da9597ee68623c536a
I20260810 15:20:35.022581 6203 tablet_bootstrap.cc:492] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Bootstrap starting.
I20260810 15:20:35.022671 6097 tablet_service.cc:1467] Processing CreateTablet for tablet f11d1d0fca224e07abb5e1ee6fa1519a (DEFAULT_TABLE table=test-table [id=5f2592329abb47cb8f32fc3b3f17c1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:35.022795 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11d1d0fca224e07abb5e1ee6fa1519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:35.022969 6203 tablet_bootstrap.cc:654] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:35.023506 6228 ts_tablet_manager.cc:1939] T 9aa47807d98d45da9597ee68623c536a P cb91403ccb8a4e9fac7cd30459df57f5: Deleting consensus metadata
I20260810 15:20:35.023600 6227 ts_tablet_manager.cc:1918] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:35.023677 6227 log.cc:1196] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/9aa47807d98d45da9597ee68623c536a
I20260810 15:20:35.023948 6227 ts_tablet_manager.cc:1939] T 9aa47807d98d45da9597ee68623c536a P 293080af495a4fe4a157d19944f07f57: Deleting consensus metadata
I20260810 15:20:35.024138 6229 tablet_replica.cc:330] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: stopping tablet replica
I20260810 15:20:35.024227 6201 tablet_bootstrap.cc:492] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Bootstrap starting.
I20260810 15:20:35.024632 6201 tablet_bootstrap.cc:654] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:35.024881 5619 catalog_manager.cc:4841] TS cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239): tablet 9aa47807d98d45da9597ee68623c536a (table test-table [id=11303d41e06d48a6b3f780b1fffb9a2c]) successfully deleted
I20260810 15:20:35.025039 5940 tablet_service.cc:1467] Processing CreateTablet for tablet f11d1d0fca224e07abb5e1ee6fa1519a (DEFAULT_TABLE table=test-table [id=5f2592329abb47cb8f32fc3b3f17c1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:35.025099 6229 raft_consensus.cc:2229] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:35.025166 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11d1d0fca224e07abb5e1ee6fa1519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:35.025246 6229 raft_consensus.cc:2258] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:35.025333 6201 tablet_bootstrap.cc:492] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: No bootstrap required, opened a new log
I20260810 15:20:35.025388 6201 ts_tablet_manager.cc:1397] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:35.025537 6201 raft_consensus.cc:348] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.025660 5619 catalog_manager.cc:4841] TS 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093): tablet 9aa47807d98d45da9597ee68623c536a (table test-table [id=11303d41e06d48a6b3f780b1fffb9a2c]) successfully deleted
I20260810 15:20:35.025607 6201 raft_consensus.cc:374] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:35.025745 6201 raft_consensus.cc:729] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a68484648e74b30b4d436ce5acc72bd, State: Initialized, Role: FOLLOWER
I20260810 15:20:35.025791 6201 consensus_queue.cc:260] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.025913 6201 ts_tablet_manager.cc:1428] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:35.026074 6229 ts_tablet_manager.cc:1905] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:35.026471 6202 tablet_bootstrap.cc:492] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Bootstrap starting.
I20260810 15:20:35.026922 6202 tablet_bootstrap.cc:654] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:35.027541 6202 tablet_bootstrap.cc:492] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: No bootstrap required, opened a new log
I20260810 15:20:35.027593 6202 ts_tablet_manager.cc:1397] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:35.027741 6202 raft_consensus.cc:348] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.027840 6202 raft_consensus.cc:374] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:35.027866 6202 raft_consensus.cc:729] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293080af495a4fe4a157d19944f07f57, State: Initialized, Role: FOLLOWER
I20260810 15:20:35.027915 6202 consensus_queue.cc:260] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.027985 6229 ts_tablet_manager.cc:1918] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:35.028024 6203 tablet_bootstrap.cc:492] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: No bootstrap required, opened a new log
I20260810 15:20:35.028024 6202 ts_tablet_manager.cc:1428] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:35.028052 6229 log.cc:1196] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/9aa47807d98d45da9597ee68623c536a
I20260810 15:20:35.028066 6203 ts_tablet_manager.cc:1397] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 15:20:35.028221 6203 raft_consensus.cc:348] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.028316 6203 raft_consensus.cc:374] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:35.028331 6229 ts_tablet_manager.cc:1939] T 9aa47807d98d45da9597ee68623c536a P 3a68484648e74b30b4d436ce5acc72bd: Deleting consensus metadata
I20260810 15:20:35.028367 6203 raft_consensus.cc:729] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb91403ccb8a4e9fac7cd30459df57f5, State: Initialized, Role: FOLLOWER
I20260810 15:20:35.028426 6203 consensus_queue.cc:260] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.028789 5619 catalog_manager.cc:4841] TS 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67:43449): tablet 9aa47807d98d45da9597ee68623c536a (table test-table [id=11303d41e06d48a6b3f780b1fffb9a2c]) successfully deleted
I20260810 15:20:35.028800 6203 ts_tablet_manager.cc:1428] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260810 15:20:35.064647 6176 tablet.cc:2369] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:35.142467 6211 raft_consensus.cc:482] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:35.142611 6211 raft_consensus.cc:504] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.142839 6211 leader_election.cc:288] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239), 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093)
I20260810 15:20:35.143074 5961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11d1d0fca224e07abb5e1ee6fa1519a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293080af495a4fe4a157d19944f07f57" is_pre_election: true
I20260810 15:20:35.143074 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11d1d0fca224e07abb5e1ee6fa1519a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" is_pre_election: true
I20260810 15:20:35.143198 5806 raft_consensus.cc:2454] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 0.
I20260810 15:20:35.143198 5961 raft_consensus.cc:2454] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 0.
I20260810 15:20:35.143378 6044 leader_election.cc:302] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a68484648e74b30b4d436ce5acc72bd, cb91403ccb8a4e9fac7cd30459df57f5; no voters:
I20260810 15:20:35.143488 6211 raft_consensus.cc:2781] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:35.143535 6211 raft_consensus.cc:482] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:35.143558 6211 raft_consensus.cc:3037] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:35.144304 6211 raft_consensus.cc:504] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.144436 6211 leader_election.cc:288] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 election: Requested vote from peers cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239), 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093)
I20260810 15:20:35.144614 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11d1d0fca224e07abb5e1ee6fa1519a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb91403ccb8a4e9fac7cd30459df57f5"
I20260810 15:20:35.144637 5961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11d1d0fca224e07abb5e1ee6fa1519a" candidate_uuid: "3a68484648e74b30b4d436ce5acc72bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "293080af495a4fe4a157d19944f07f57"
I20260810 15:20:35.144703 5961 raft_consensus.cc:3037] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:35.144717 5806 raft_consensus.cc:3037] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:35.145409 5806 raft_consensus.cc:2454] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 1.
I20260810 15:20:35.145673 6044 leader_election.cc:302] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a68484648e74b30b4d436ce5acc72bd, cb91403ccb8a4e9fac7cd30459df57f5; no voters:
I20260810 15:20:35.145689 5961 raft_consensus.cc:2454] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a68484648e74b30b4d436ce5acc72bd in term 1.
I20260810 15:20:35.145792 6211 raft_consensus.cc:2781] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:35.145836 6211 raft_consensus.cc:686] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 1 LEADER]: Becoming Leader. State: Replica: 3a68484648e74b30b4d436ce5acc72bd, State: Running, Role: LEADER
I20260810 15:20:35.145890 6211 consensus_queue.cc:237] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.146533 5632 catalog_manager.cc:5495] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67). New cstate: current_term: 1 leader_uuid: "3a68484648e74b30b4d436ce5acc72bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } health_report { overall_health: HEALTHY } } }
Aug 10 15:20:35 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:35.194202 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.186616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.4.64.126:36923)
0810 15:20:35.186683 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.186685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.186898 (+ 213us) server_negotiation.cc:407] Connection header received
0810 15:20:35.186979 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.186981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.186992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.187069 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:35.187715 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.188230 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.189011 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.189204 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.189281 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.191458 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:35.191459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:35.191459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:35.191467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:35.192748 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:35.193138 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:35.193139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:35.193139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:35.193210 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:35.193418 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:35.193419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:35.193419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:35.193559 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:35.193656 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.193823 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.194050 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:20:35.194307 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.186551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36923 (local address 127.0.0.1:33080)
0810 15:20:35.186774 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:35.186785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:35.186923 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:35.187074 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:35.187075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:35.187557 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:35.187623 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.187626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.188312 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.188314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.188923 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.188925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.189279 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.189279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.189331 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.189956 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:35.189960 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:35.191365 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:35.192853 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:35.192862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:35.192864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:35.193064 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:35.193279 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:35.193280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:35.193280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:35.193346 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:35.193641 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:35.193649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:35.193699 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:20:35.193799 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 15:20:35.194157 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":156,"thread_start_us":137,"threads_started":1}
I20260810 15:20:35.196995 5632 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:33080:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 15:20:35.197196 5632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:35.199146 5786 tablet_service.cc:1467] Processing CreateTablet for tablet e259c9820dd241209d5d4110a30354e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c7fb3edb17b4b639f74d7f4def5a35a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:20:35.199157 5940 tablet_service.cc:1467] Processing CreateTablet for tablet e259c9820dd241209d5d4110a30354e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c7fb3edb17b4b639f74d7f4def5a35a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:20:35.199128 6097 tablet_service.cc:1467] Processing CreateTablet for tablet e259c9820dd241209d5d4110a30354e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4c7fb3edb17b4b639f74d7f4def5a35a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:20:35.199321 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259c9820dd241209d5d4110a30354e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:35.199321 6097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259c9820dd241209d5d4110a30354e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:35.199334 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e259c9820dd241209d5d4110a30354e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:35.200555 6203 tablet_bootstrap.cc:492] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: Bootstrap starting.
I20260810 15:20:35.200907 6201 tablet_bootstrap.cc:492] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: Bootstrap starting.
I20260810 15:20:35.200960 6203 tablet_bootstrap.cc:654] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:35.201197 6201 tablet_bootstrap.cc:654] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:35.201516 6202 tablet_bootstrap.cc:492] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: Bootstrap starting.
I20260810 15:20:35.201833 6203 tablet_bootstrap.cc:492] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: No bootstrap required, opened a new log
I20260810 15:20:35.201877 6203 ts_tablet_manager.cc:1397] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:35.201943 6202 tablet_bootstrap.cc:654] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:35.201993 6201 tablet_bootstrap.cc:492] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: No bootstrap required, opened a new log
I20260810 15:20:35.202006 6203 raft_consensus.cc:348] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.202041 6201 ts_tablet_manager.cc:1397] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:35.202062 6203 raft_consensus.cc:374] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:35.202076 6203 raft_consensus.cc:729] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb91403ccb8a4e9fac7cd30459df57f5, State: Initialized, Role: FOLLOWER
I20260810 15:20:35.202139 6203 consensus_queue.cc:260] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.202167 6201 raft_consensus.cc:348] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.202240 6201 raft_consensus.cc:374] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:35.202263 6201 raft_consensus.cc:729] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a68484648e74b30b4d436ce5acc72bd, State: Initialized, Role: FOLLOWER
I20260810 15:20:35.202257 6203 ts_tablet_manager.cc:1428] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:35.202312 6201 consensus_queue.cc:260] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.202397 6201 ts_tablet_manager.cc:1428] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:35.202389 6208 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.202499 6208 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.202504 6211 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.202581 6211 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.203305 6202 tablet_bootstrap.cc:492] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: No bootstrap required, opened a new log
I20260810 15:20:35.203361 6202 ts_tablet_manager.cc:1397] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:35.203498 6202 raft_consensus.cc:348] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.203547 6202 raft_consensus.cc:374] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:35.203558 6202 raft_consensus.cc:729] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293080af495a4fe4a157d19944f07f57, State: Initialized, Role: FOLLOWER
I20260810 15:20:35.203594 6202 consensus_queue.cc:260] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.203653 6202 ts_tablet_manager.cc:1428] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:35.203794 6209 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.203897 6209 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.210662 6209 raft_consensus.cc:482] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:35.210783 6209 raft_consensus.cc:504] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.211153 6209 leader_election.cc:288] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239), 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67:43449)
I20260810 15:20:35.214583 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.211409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:38667 (local address 127.4.64.65:43239)
0810 15:20:35.211474 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.211476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.211612 (+ 136us) server_negotiation.cc:407] Connection header received
0810 15:20:35.211677 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.211678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.211691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.211793 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:35.211979 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.212553 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.213758 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.214214 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.214253 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.214263 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.214460 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.214515 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:20:35.214596 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.211347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:43239 (local address 127.4.64.66:38667)
0810 15:20:35.211508 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:35.211519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:35.211613 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:35.211801 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:35.211801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:35.211804 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:35.211895 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.211898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.212654 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.212654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.213602 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.213648 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.214335 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.214336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.214388 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.214391 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:35.214430 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 15:20:35.214461 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":24,"thread_start_us":99,"threads_started":1}
I20260810 15:20:35.214679 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.211674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:46963 (local address 127.4.64.67:43449)
0810 15:20:35.211723 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.211726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.211735 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:20:35.211787 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.211788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.211797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.211851 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:35.212055 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.212582 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.213753 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.214215 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.214316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.214329 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.214546 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.214601 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:20:35.214865 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.211560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43449 (local address 127.4.64.66:46963)
0810 15:20:35.211671 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:35.211681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:35.211731 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:35.211865 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:35.211866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:35.211868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:35.211977 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.211979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.212702 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.212702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.213628 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.213673 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.214346 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.214348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.214450 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.214455 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:20:35.214518 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:20:35.214556 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":74,"thread_start_us":59,"threads_started":1}
I20260810 15:20:35.215049 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259c9820dd241209d5d4110a30354e8" candidate_uuid: "293080af495a4fe4a157d19944f07f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" is_pre_election: true
I20260810 15:20:35.215150 5806 raft_consensus.cc:2454] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293080af495a4fe4a157d19944f07f57 in term 0.
I20260810 15:20:35.215198 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259c9820dd241209d5d4110a30354e8" candidate_uuid: "293080af495a4fe4a157d19944f07f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a68484648e74b30b4d436ce5acc72bd" is_pre_election: true
I20260810 15:20:35.215340 6121 raft_consensus.cc:2454] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 293080af495a4fe4a157d19944f07f57 in term 0.
I20260810 15:20:35.215312 5892 leader_election.cc:302] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293080af495a4fe4a157d19944f07f57, cb91403ccb8a4e9fac7cd30459df57f5; no voters:
I20260810 15:20:35.215487 6209 raft_consensus.cc:2781] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:35.215539 6209 raft_consensus.cc:482] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:35.215572 6209 raft_consensus.cc:3037] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:35.216392 6209 raft_consensus.cc:504] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.216552 6209 leader_election.cc:288] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [CANDIDATE]: Term 1 election: Requested vote from peers cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239), 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67:43449)
I20260810 15:20:35.216770 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259c9820dd241209d5d4110a30354e8" candidate_uuid: "293080af495a4fe4a157d19944f07f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a68484648e74b30b4d436ce5acc72bd"
I20260810 15:20:35.216768 5806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e259c9820dd241209d5d4110a30354e8" candidate_uuid: "293080af495a4fe4a157d19944f07f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb91403ccb8a4e9fac7cd30459df57f5"
I20260810 15:20:35.216854 6121 raft_consensus.cc:3037] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:35.216854 5806 raft_consensus.cc:3037] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:35.217675 5806 raft_consensus.cc:2454] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293080af495a4fe4a157d19944f07f57 in term 1.
I20260810 15:20:35.217777 6121 raft_consensus.cc:2454] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 293080af495a4fe4a157d19944f07f57 in term 1.
I20260810 15:20:35.217857 5892 leader_election.cc:302] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 293080af495a4fe4a157d19944f07f57, cb91403ccb8a4e9fac7cd30459df57f5; no voters:
I20260810 15:20:35.217971 6209 raft_consensus.cc:2781] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:35.218115 6209 raft_consensus.cc:686] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [term 1 LEADER]: Becoming Leader. State: Replica: 293080af495a4fe4a157d19944f07f57, State: Running, Role: LEADER
I20260810 15:20:35.218187 6209 consensus_queue.cc:237] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } }
I20260810 15:20:35.218848 6248 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: TxnStatusTablet state changed. Reason: New leader 293080af495a4fe4a157d19944f07f57. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.218986 6248 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:20:35.219086 5631 catalog_manager.cc:5495] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 293080af495a4fe4a157d19944f07f57 (127.4.64.66). New cstate: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } health_report { overall_health: UNKNOWN } } }
I20260810 15:20:35.219272 6250 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 15:20:35.219411 6250 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 15:20:35.229099 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.226093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.4.64.66:37093)
0810 15:20:35.226142 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.226144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.226195 (+ 51us) server_negotiation.cc:407] Connection header received
0810 15:20:35.226302 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.226303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.226319 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.226378 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:35.226516 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.226984 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.228238 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.228423 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.228479 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.228747 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.228856 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.228920 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.229020 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.229041 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:35.229185 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.226036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:37093 (local address 127.0.0.1:60832)
0810 15:20:35.226145 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:35.226158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:35.226218 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:35.226386 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:35.226387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:35.226389 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:35.226452 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.226455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.227075 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.227076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.228124 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.228128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.228515 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.228516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.228601 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.228623 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.228936 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.228947 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 15:20:35.229010 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:20:35.229075 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 15:20:35.230598 5806 raft_consensus.cc:1261] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Refusing update from remote peer 293080af495a4fe4a157d19944f07f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:35.230611 6121 raft_consensus.cc:1261] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Refusing update from remote peer 293080af495a4fe4a157d19944f07f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:35.230964 6248 consensus_queue.cc:1035] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:35.231106 6209 consensus_queue.cc:1035] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:35.231690 6208 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: TxnStatusTablet state changed. Reason: New leader 293080af495a4fe4a157d19944f07f57. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.231783 6208 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.231933 6211 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: TxnStatusTablet state changed. Reason: New leader 293080af495a4fe4a157d19944f07f57. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.232035 6211 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.232759 6209 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.232842 6209 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:20:35.233174 6215 mvcc.cc:205] Tried to move back new op lower bound from 7316992963502034944 to 7316992963454541824. Current Snapshot: MvccSnapshot[applied={T|T < 7316992963502034944}]
I20260810 15:20:35.233259 6211 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.233322 6211 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 3a68484648e74b30b4d436ce5acc72bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.233664 6255 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.233764 6255 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P 293080af495a4fe4a157d19944f07f57: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:20:35.233894 6208 tablet_replica.cc:439] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "293080af495a4fe4a157d19944f07f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 } } peers { permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 } } peers { permanent_uuid: "3a68484648e74b30b4d436ce5acc72bd" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43449 } } }
I20260810 15:20:35.233980 6208 tablet_replica.cc:442] T e259c9820dd241209d5d4110a30354e8 P cb91403ccb8a4e9fac7cd30459df57f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:35.240244 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.237421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:37093 (local address 127.4.64.67:59765)
0810 15:20:35.237499 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:35.237512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:35.237568 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:35.237735 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:35.237736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:35.237738 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:35.237792 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.237794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.238409 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.238410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.239390 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.239392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.239752 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.239753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.239819 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.239842 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.240026 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.240027 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:20:35.240079 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:20:35.240133 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 15:20:35.240290 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.237532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:59765 (local address 127.4.64.66:37093)
0810 15:20:35.237592 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.237594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.237642 (+ 48us) server_negotiation.cc:407] Connection header received
0810 15:20:35.237665 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.237666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.237677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.237727 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:35.237862 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.238337 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.239490 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.239677 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.239729 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.239931 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.239961 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.240004 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.240120 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.240146 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:20:35.243047 5806 raft_consensus.cc:1261] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Refusing update from remote peer 3a68484648e74b30b4d436ce5acc72bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:35.243115 5961 raft_consensus.cc:1261] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Refusing update from remote peer 3a68484648e74b30b4d436ce5acc72bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:35.243319 6221 consensus_queue.cc:1035] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb91403ccb8a4e9fac7cd30459df57f5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:35.243443 6211 consensus_queue.cc:1035] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "293080af495a4fe4a157d19944f07f57" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:35.253762 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.250782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:48443 (local address 127.4.64.67:43449)
0810 15:20:35.250843 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.250845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.251033 (+ 188us) server_negotiation.cc:407] Connection header received
0810 15:20:35.251073 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.251074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.251089 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.251157 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:35.251313 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.251840 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.253032 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.253223 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.253279 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.253478 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.253505 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.253539 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.253663 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.253687 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:20:35.253829 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.250671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43449 (local address 127.4.64.66:48443)
0810 15:20:35.250931 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:35.250943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:35.251015 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:35.251164 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:35.251165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:35.251166 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:35.251248 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.251250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.251919 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.251920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.252940 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:35.252943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.253301 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.253302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:35.253375 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.253395 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:35.253556 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:35.253557 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:20:35.253609 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:20:35.253719 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":133,"threads_started":1}
I20260810 15:20:35.262725 5632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:35.262861 5632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:35.263538 5632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cec9a435fbef4dc1bfac66e17d5939e8: Sending DeleteTablet for 3 replicas of tablet f11d1d0fca224e07abb5e1ee6fa1519a
I20260810 15:20:35.263841 5940 tablet_service.cc:1514] Processing DeleteTablet for tablet f11d1d0fca224e07abb5e1ee6fa1519a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:35 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:60820
I20260810 15:20:35.263878 6097 tablet_service.cc:1514] Processing DeleteTablet for tablet f11d1d0fca224e07abb5e1ee6fa1519a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:35 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:46226
I20260810 15:20:35.264021 5786 tablet_service.cc:1514] Processing DeleteTablet for tablet f11d1d0fca224e07abb5e1ee6fa1519a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:35 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47226
I20260810 15:20:35.264112 6228 tablet_replica.cc:330] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: stopping tablet replica
I20260810 15:20:35.264168 6228 raft_consensus.cc:2229] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:35.264220 6228 raft_consensus.cc:2258] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:35.264402 6227 tablet_replica.cc:330] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: stopping tablet replica
I20260810 15:20:35.264478 6227 raft_consensus.cc:2229] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:35.264516 6227 raft_consensus.cc:2258] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:35.264593 6229 tablet_replica.cc:330] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: stopping tablet replica
I20260810 15:20:35.264657 6229 raft_consensus.cc:2229] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:35.264729 6229 raft_consensus.cc:2258] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:35.265038 6227 ts_tablet_manager.cc:1905] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:35.265178 6228 ts_tablet_manager.cc:1905] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:35.267058 6229 ts_tablet_manager.cc:1905] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:35.267345 6228 ts_tablet_manager.cc:1918] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:35.267334 6227 ts_tablet_manager.cc:1918] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:35.267408 6227 log.cc:1196] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/f11d1d0fca224e07abb5e1ee6fa1519a
I20260810 15:20:35.267409 6228 log.cc:1196] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/f11d1d0fca224e07abb5e1ee6fa1519a
I20260810 15:20:35.267644 6227 ts_tablet_manager.cc:1939] T f11d1d0fca224e07abb5e1ee6fa1519a P 293080af495a4fe4a157d19944f07f57: Deleting consensus metadata
I20260810 15:20:35.267653 6228 ts_tablet_manager.cc:1939] T f11d1d0fca224e07abb5e1ee6fa1519a P cb91403ccb8a4e9fac7cd30459df57f5: Deleting consensus metadata
I20260810 15:20:35.267988 5619 catalog_manager.cc:4841] TS cb91403ccb8a4e9fac7cd30459df57f5 (127.4.64.65:43239): tablet f11d1d0fca224e07abb5e1ee6fa1519a (table test-table [id=5f2592329abb47cb8f32fc3b3f17c1ac]) successfully deleted
I20260810 15:20:35.268083 5619 catalog_manager.cc:4841] TS 293080af495a4fe4a157d19944f07f57 (127.4.64.66:37093): tablet f11d1d0fca224e07abb5e1ee6fa1519a (table test-table [id=5f2592329abb47cb8f32fc3b3f17c1ac]) successfully deleted
I20260810 15:20:35.268725 6229 ts_tablet_manager.cc:1918] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:35.268795 6229 log.cc:1196] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/f11d1d0fca224e07abb5e1ee6fa1519a
I20260810 15:20:35.269019 6229 ts_tablet_manager.cc:1939] T f11d1d0fca224e07abb5e1ee6fa1519a P 3a68484648e74b30b4d436ce5acc72bd: Deleting consensus metadata
I20260810 15:20:35.269522 5619 catalog_manager.cc:4841] TS 3a68484648e74b30b4d436ce5acc72bd (127.4.64.67:43449): tablet f11d1d0fca224e07abb5e1ee6fa1519a (table test-table [id=5f2592329abb47cb8f32fc3b3f17c1ac]) successfully deleted
Aug 10 15:20:35 dist-test-slave-9zdj krb5kdc[5583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:35.275951 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.264556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.4.64.65:43239)
0810 15:20:35.264672 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.264675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.264945 (+ 270us) server_negotiation.cc:407] Connection header received
0810 15:20:35.265064 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.265065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.265081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.265113 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:35.266051 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.266775 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.268884 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.269207 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.269312 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.272765 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:35.272778 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:35.272779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:35.272788 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:35.274627 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:35.275019 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:35.275023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:35.275024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:35.275112 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:35.275334 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:35.275335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:35.275336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:35.275505 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:35.275672 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.275747 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.275839 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 15:20:35.276312 5851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47242
I20260810 15:20:35.283851 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:35.277259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.4.64.126:36923)
0810 15:20:35.277308 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:35.277310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:35.277827 (+ 517us) server_negotiation.cc:407] Connection header received
0810 15:20:35.277915 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:35.277916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:35.277928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:35.277979 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:35.278542 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.279047 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.279744 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:35.279964 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:35.280043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:35.281229 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:35.281230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:35.281231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:35.281238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:35.282684 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:35.283039 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:35.283040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:35.283041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:35.283106 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:35.283321 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:35.283322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:35.283323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:35.283445 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:35.283550 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:35.283658 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 15:20:35.283738 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 15:20:35.284096 5632 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33096
I20260810 15:20:35.284808 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 5712
I20260810 15:20:35.294559 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 5868
I20260810 15:20:35.304080 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 6022
I20260810 15:20:35.312685 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 5597
2026-08-10T15:20:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2448 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:35 dist-test-slave-9zdj krb5kdc[6270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:35 dist-test-slave-9zdj krb5kdc[6270](info): set up 2 sockets
Aug 10 15:20:35 dist-test-slave-9zdj krb5kdc[6270](info): commencing operation
krb5kdc: starting...
W20260810 15:20:36.432962 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:36 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:36.577092 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:45447
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:45447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:36.661800 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:36.662014 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:36.662035 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:36.663199 6285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:36.663244 6285 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 15:20:36.663257 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:36.663270 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:36.663282 6285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:36.663295 6285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:36.664806 6285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:45447
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:45447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:36.665061 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:36.665321 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:36.670398 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:36.670408 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:36.670462 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:36.670768 6285 server_base.cc:1034] running on GCE node
I20260810 15:20:36.671036 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:36.671375 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:36.672550 6285 hybrid_clock.cc:648] HybridClock initialized: now 1786375236672525 us; error 41 us; skew 500 ppm
Aug 10 15:20:36 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375236, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:36.675330 6285 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:36.675910 6285 webserver.cc:460] Webserver started at http://127.4.64.126:39015/ using document root <none> and password file <none>
I20260810 15:20:36.676160 6285 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:36.676234 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:36.676399 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:36.677536 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "210407ab55da425c955abb8552f5db87"
format_stamp: "Formatted at 2026-08-10 15:20:36 on dist-test-slave-9zdj"
server_key: "063de94fc9248cac61431b4b4ad6ed7d"
server_key_iv: "b4082c3a45484185138f80623ad6c871"
server_key_version: "encryptionkey@0"
I20260810 15:20:36.678123 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "210407ab55da425c955abb8552f5db87"
format_stamp: "Formatted at 2026-08-10 15:20:36 on dist-test-slave-9zdj"
server_key: "063de94fc9248cac61431b4b4ad6ed7d"
server_key_iv: "b4082c3a45484185138f80623ad6c871"
server_key_version: "encryptionkey@0"
I20260810 15:20:36.679692 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:36.680591 6301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:36.680807 6285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:36.680950 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "210407ab55da425c955abb8552f5db87"
format_stamp: "Formatted at 2026-08-10 15:20:36 on dist-test-slave-9zdj"
server_key: "063de94fc9248cac61431b4b4ad6ed7d"
server_key_iv: "b4082c3a45484185138f80623ad6c871"
server_key_version: "encryptionkey@0"
I20260810 15:20:36.681043 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:36.707187 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:36.709329 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:36.709506 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:36.714684 6285 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:45447
I20260810 15:20:36.714738 6371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:45447 every 8 connection(s)
I20260810 15:20:36.715104 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:36.715806 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:36.718204 6372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Bootstrap starting.
I20260810 15:20:36.718781 6372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:36.719018 6372 log.cc:824] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:36.719692 6372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: No bootstrap required, opened a new log
I20260810 15:20:36.721979 6372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210407ab55da425c955abb8552f5db87" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45447 } }
I20260810 15:20:36.722127 6372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:36.722149 6372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210407ab55da425c955abb8552f5db87, State: Initialized, Role: FOLLOWER
I20260810 15:20:36.722292 6372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210407ab55da425c955abb8552f5db87" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45447 } }
I20260810 15:20:36.722397 6372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:36.722441 6372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:36.722504 6372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:36.723409 6372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210407ab55da425c955abb8552f5db87" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45447 } }
I20260810 15:20:36.723543 6372 leader_election.cc:302] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 210407ab55da425c955abb8552f5db87; no voters:
I20260810 15:20:36.723685 6372 leader_election.cc:288] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:36.723795 6375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:36.723985 6372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:36.723991 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 6285
I20260810 15:20:36.724102 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:36.724067 6375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [term 1 LEADER]: Becoming Leader. State: Replica: 210407ab55da425c955abb8552f5db87, State: Running, Role: LEADER
I20260810 15:20:36.724298 4353 external_mini_cluster.cc:1428] Setting key 2c17c365e30ea6864b69316160fcc757
I20260810 15:20:36.724326 6375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210407ab55da425c955abb8552f5db87" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45447 } }
I20260810 15:20:36.724758 6377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 210407ab55da425c955abb8552f5db87. Latest consensus state: current_term: 1 leader_uuid: "210407ab55da425c955abb8552f5db87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210407ab55da425c955abb8552f5db87" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45447 } } }
I20260810 15:20:36.724862 6377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:36.725792 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "210407ab55da425c955abb8552f5db87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "210407ab55da425c955abb8552f5db87" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45447 } } }
I20260810 15:20:36.725939 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:36.726343 6393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:36.726403 6393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:36.726449 6385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:36.726614 6385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:36.728293 6385 catalog_manager.cc:1347] Generated new cluster ID: 1b9e9b67f82444bca069d9a226dccae6
I20260810 15:20:36.728350 6385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:20:36 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:36.737381 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:36.725015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.4.64.126:45447)
0810 15:20:36.725906 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:36.725917 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:36.726189 (+ 272us) server_negotiation.cc:407] Connection header received
0810 15:20:36.726336 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:36.726342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:36.726388 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:36.726550 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:36.727357 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.728766 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.729850 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.730463 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.730550 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:36.733441 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:36.733445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:36.733446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:36.733458 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:36.735458 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:36.735847 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:36.735851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:36.735851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:36.735956 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:36.736122 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:36.736123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:36.736123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:36.736413 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:36.736556 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:36.736995 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 15:20:36.737138 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":81,"threads_started":1}
I20260810 15:20:36.740078 6385 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:36.740792 6385 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:36.748771 6385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Generated new TSK 0
I20260810 15:20:36.749058 6385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:36.785102 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45447
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:36.878700 6399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:36.878952 6399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:36.878988 6399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:36.880345 6399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:36.880394 6399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:36.880417 6399 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 15:20:36.880429 6399 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:20:36.880451 6399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:36.882287 6399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:45447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:36.882545 6399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:36.882779 6399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:36.888813 6407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:36.888895 6405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:36.888813 6404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:36.889021 6399 server_base.cc:1034] running on GCE node
I20260810 15:20:36.889382 6399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:36.889744 6399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:36.890947 6399 hybrid_clock.cc:648] HybridClock initialized: now 1786375236890883 us; error 82 us; skew 500 ppm
Aug 10 15:20:36 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375236, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:36.894141 6399 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:36.894680 6399 webserver.cc:460] Webserver started at http://127.4.64.65:33411/ using document root <none> and password file <none>
I20260810 15:20:36.894996 6399 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:36.895083 6399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:36.895258 6399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:36.896531 6399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "c657f2357768432c9a5f3258ebfe1108"
format_stamp: "Formatted at 2026-08-10 15:20:36 on dist-test-slave-9zdj"
server_key: "81682dca880f20485a76f95514e5af33"
server_key_iv: "2bbaafa9c770129c8dde922d1b938fbb"
server_key_version: "encryptionkey@0"
I20260810 15:20:36.896965 6399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "c657f2357768432c9a5f3258ebfe1108"
format_stamp: "Formatted at 2026-08-10 15:20:36 on dist-test-slave-9zdj"
server_key: "81682dca880f20485a76f95514e5af33"
server_key_iv: "2bbaafa9c770129c8dde922d1b938fbb"
server_key_version: "encryptionkey@0"
I20260810 15:20:36.898736 6399 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 15:20:36.899724 6415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:36.899996 6399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:36.900126 6399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "c657f2357768432c9a5f3258ebfe1108"
format_stamp: "Formatted at 2026-08-10 15:20:36 on dist-test-slave-9zdj"
server_key: "81682dca880f20485a76f95514e5af33"
server_key_iv: "2bbaafa9c770129c8dde922d1b938fbb"
server_key_version: "encryptionkey@0"
I20260810 15:20:36.900218 6399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:36.941581 6399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:36.944237 6399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:36.944430 6399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:36.944983 6399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:36.945046 6399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:36.945081 6399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:36.945112 6399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:36.952437 6399 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:45955
I20260810 15:20:36.952518 6543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:45955 every 8 connection(s)
Aug 10 15:20:36 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375236, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:36.952862 6399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:36.956051 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:36.945770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:51093 (local address 127.4.64.126:45447)
0810 15:20:36.945830 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:36.945836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:36.946110 (+ 274us) server_negotiation.cc:407] Connection header received
0810 15:20:36.946708 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:36.946709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:36.946725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:36.946783 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:36.947724 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.948608 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.950364 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.950732 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.950827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:36.953489 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:36.953491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:36.953491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:36.953502 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:36.954802 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:36.955183 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:36.955184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:36.955184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:36.955241 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:36.955433 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:36.955434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:36.955434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:36.955543 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:36.955597 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:36.955901 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 15:20:36.955985 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:20:36.956138 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:36.945785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.4.64.65:51093)
0810 15:20:36.945981 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:36.946020 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:36.946598 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:36.946874 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:36.946879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:36.947290 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:36.947595 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:36.947601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.948728 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.948729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:36.950246 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:36.950250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.950834 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.950835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:36.950983 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:36.951599 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:36.951607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:36.953374 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:36.954882 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:36.954889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:36.954890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:36.955112 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:36.955290 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:36.955291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:36.955291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:36.955381 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:36.955612 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:36.955616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:36.955739 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 15:20:36.955827 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 15:20:36.955963 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260810 15:20:36.956703 6544 heartbeater.cc:347] Connected to a master server at 127.4.64.126:45447
I20260810 15:20:36.956800 6544 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:36.956995 6544 heartbeater.cc:511] Master 127.4.64.126:45447 requested a full tablet report, sending...
I20260810 15:20:36.957656 6319 ts_manager.cc:194] Registered new tserver with Master: c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955)
I20260810 15:20:36.958546 6319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:51093
I20260810 15:20:36.962567 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 6399
I20260810 15:20:36.962695 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:36.962893 4353 external_mini_cluster.cc:1428] Setting key ab4207e0a2250a62705cd37f3ecf8519
I20260810 15:20:36.964049 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:36.957964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:42361 (local address 127.4.64.126:45447)
0810 15:20:36.958010 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:36.958012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:36.958142 (+ 130us) server_negotiation.cc:407] Connection header received
0810 15:20:36.958167 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:36.958168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:36.958178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:36.958248 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:36.958523 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.959175 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.959895 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.960083 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.960152 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:36.961089 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:36.961091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:36.961091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:36.961098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:36.962663 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:36.963151 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:36.963152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:36.963152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:36.963219 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:36.963429 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:36.963430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:36.963431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:36.963566 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:36.963637 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:36.963891 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 15:20:36.963981 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:36.964130 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:36.957897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.4.64.65:42361)
0810 15:20:36.958070 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:36.958080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:36.958130 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:36.958249 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:36.958250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:36.958407 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:36.958459 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:36.958461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.959301 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.959302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:36.959814 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:36.959816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:36.960155 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:36.960155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:36.960214 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:36.960701 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:36.960706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:36.960988 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:36.962797 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:36.962799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:36.962799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:36.963045 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:36.963295 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:36.963296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:36.963296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:36.963360 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:36.963694 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:36.963695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:36.963760 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 15:20:36.963838 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 15:20:36.963991 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:36.993130 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45447
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:37.081982 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:37.082218 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:37.082262 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:37.083442 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:37.083510 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:37.083572 6553 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 15:20:37.083618 6553 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:20:37.083652 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:37.085225 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:45447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:37.085506 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:37.085799 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:37.091102 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:37.091150 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:37.091136 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:37.091205 6553 server_base.cc:1034] running on GCE node
I20260810 15:20:37.091564 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:37.091851 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:37.093052 6553 hybrid_clock.cc:648] HybridClock initialized: now 1786375237093030 us; error 38 us; skew 500 ppm
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375237, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:37.095660 6553 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:37.096088 6553 webserver.cc:460] Webserver started at http://127.4.64.66:43931/ using document root <none> and password file <none>
I20260810 15:20:37.096302 6553 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:37.096344 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:37.096436 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:37.097450 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
format_stamp: "Formatted at 2026-08-10 15:20:37 on dist-test-slave-9zdj"
server_key: "a5559740d08569ce92a1a5598c7c22e8"
server_key_iv: "5186d3eab82e0f23ca8145335154ca58"
server_key_version: "encryptionkey@0"
I20260810 15:20:37.097856 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
format_stamp: "Formatted at 2026-08-10 15:20:37 on dist-test-slave-9zdj"
server_key: "a5559740d08569ce92a1a5598c7c22e8"
server_key_iv: "5186d3eab82e0f23ca8145335154ca58"
server_key_version: "encryptionkey@0"
I20260810 15:20:37.099360 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:37.100181 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.100489 6553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:37.100576 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
format_stamp: "Formatted at 2026-08-10 15:20:37 on dist-test-slave-9zdj"
server_key: "a5559740d08569ce92a1a5598c7c22e8"
server_key_iv: "5186d3eab82e0f23ca8145335154ca58"
server_key_version: "encryptionkey@0"
I20260810 15:20:37.100634 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:37.119727 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:37.122011 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:37.122148 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:37.122696 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:37.122740 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.122766 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:37.122779 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.130378 6553 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:39655
I20260810 15:20:37.130425 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:39655 every 8 connection(s)
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375237, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:37.130770 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:37.133901 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.123331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:51665 (local address 127.4.64.126:45447)
0810 15:20:37.123384 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.123386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.123675 (+ 289us) server_negotiation.cc:407] Connection header received
0810 15:20:37.124243 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.124245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.124258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.124343 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:37.125372 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.126304 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.127965 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.128293 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.128384 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.131330 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:37.131332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:37.131332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:37.131343 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:37.132604 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.132992 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.132992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.132993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.133048 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.133266 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.133266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.133267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.133376 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:37.133428 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.133746 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.133827 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:37.134092 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.123308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.4.64.66:51665)
0810 15:20:37.123509 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.123546 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.124126 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.124403 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.124408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.124895 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:37.125228 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.125235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.126415 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.126416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.127863 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.127867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.128390 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.128390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.128503 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.129141 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:37.129153 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:37.131224 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:37.132681 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.132688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.132690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.132919 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.133107 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.133108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.133108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.133211 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.133433 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:37.133437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:37.133559 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.133716 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.133853 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260810 15:20:37.134641 6700 heartbeater.cc:347] Connected to a master server at 127.4.64.126:45447
I20260810 15:20:37.134732 6700 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:37.134948 6700 heartbeater.cc:511] Master 127.4.64.126:45447 requested a full tablet report, sending...
I20260810 15:20:37.135365 6320 ts_manager.cc:194] Registered new tserver with Master: d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.135767 6320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:51665
I20260810 15:20:37.139624 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 6553
I20260810 15:20:37.139739 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:37.139930 4353 external_mini_cluster.cc:1428] Setting key 8f7fbd6afaaf43e4b88b8f73a65608c2
I20260810 15:20:37.141423 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.135817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:49813 (local address 127.4.64.126:45447)
0810 15:20:37.135870 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.135872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.136017 (+ 145us) server_negotiation.cc:407] Connection header received
0810 15:20:37.136060 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.136061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.136071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.136133 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:37.136499 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.136964 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.137646 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.137870 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.137924 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.138754 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:37.138756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:37.138756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:37.138762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:37.140360 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.140649 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.140650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.140650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.140713 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.140942 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.140943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.140943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.141086 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:37.141140 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.141300 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.141362 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:20:37.141850 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.135745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.4.64.66:49813)
0810 15:20:37.135945 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.135956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.136012 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.136138 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.136139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.136360 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:37.136425 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.136427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.137051 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.137052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.137544 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.137546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.137936 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.137936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.137990 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.138395 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:37.138399 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:37.138678 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:37.140431 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.140431 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.140432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.140594 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.140799 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.140800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.140801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.140874 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.141153 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:37.141154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:37.141204 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.141285 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.141411 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:37.173449 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45447
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:37.266052 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:37.266268 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:37.266294 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:37.267405 6710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:37.267439 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:37.267459 6710 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 15:20:37.267472 6710 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:20:37.267488 6710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:37.268889 6710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:45447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:37.269088 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:37.269271 6710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:37.274652 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:37.274629 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:37.274657 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:37.274925 6710 server_base.cc:1034] running on GCE node
I20260810 15:20:37.275118 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:37.275352 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:37.276503 6710 hybrid_clock.cc:648] HybridClock initialized: now 1786375237276490 us; error 26 us; skew 500 ppm
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375237, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:37.283857 6710 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:37.284327 6710 webserver.cc:460] Webserver started at http://127.4.64.67:45535/ using document root <none> and password file <none>
I20260810 15:20:37.284603 6710 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:37.284682 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:37.284833 6710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:37.286015 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc"
format_stamp: "Formatted at 2026-08-10 15:20:37 on dist-test-slave-9zdj"
server_key: "f8648e86e86682c9b17274faebc5b34c"
server_key_iv: "85f4a66fbff288e8831f49c537df48df"
server_key_version: "encryptionkey@0"
I20260810 15:20:37.286420 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc"
format_stamp: "Formatted at 2026-08-10 15:20:37 on dist-test-slave-9zdj"
server_key: "f8648e86e86682c9b17274faebc5b34c"
server_key_iv: "85f4a66fbff288e8831f49c537df48df"
server_key_version: "encryptionkey@0"
I20260810 15:20:37.287794 6710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:37.288563 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.288784 6710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:37.288915 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc"
format_stamp: "Formatted at 2026-08-10 15:20:37 on dist-test-slave-9zdj"
server_key: "f8648e86e86682c9b17274faebc5b34c"
server_key_iv: "85f4a66fbff288e8831f49c537df48df"
server_key_version: "encryptionkey@0"
I20260810 15:20:37.288993 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:37.308501 6710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:37.310520 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:37.310720 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:37.311298 6710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:37.311373 6710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.311419 6710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:37.311448 6710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375237, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:37.318849 6710 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43899
I20260810 15:20:37.318919 6856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43899 every 8 connection(s)
I20260810 15:20:37.319314 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:37.321009 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 6710
I20260810 15:20:37.321122 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:37.321398 4353 external_mini_cluster.cc:1428] Setting key d24ea4acc24ca8e39b585ed0c1ef9966
I20260810 15:20:37.322167 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.311871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:48303 (local address 127.4.64.126:45447)
0810 15:20:37.311947 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.311951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.312200 (+ 249us) server_negotiation.cc:407] Connection header received
0810 15:20:37.312714 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.312716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.312731 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.312817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:37.313782 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.314308 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.316001 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.316378 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.316470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.319030 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:37.319032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:37.319032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:37.319041 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:37.320544 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.320976 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.320977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.320977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.321051 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.321365 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.321366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.321366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.321527 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:37.321608 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.321978 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.322068 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:20:37.322422 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.311858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.4.64.67:48303)
0810 15:20:37.312035 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.312075 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.312608 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.312888 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.312892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.313334 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:37.313681 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.313687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.314409 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.314410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.315888 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.315891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.316492 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.316493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.316666 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.317369 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:37.317377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:37.318933 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:37.320624 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.320630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.320631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.320878 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.321140 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.321141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.321141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.321286 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.321605 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:37.321628 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:37.321813 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.321941 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.322136 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260810 15:20:37.322876 6857 heartbeater.cc:347] Connected to a master server at 127.4.64.126:45447
I20260810 15:20:37.322960 6857 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:37.323158 6857 heartbeater.cc:511] Master 127.4.64.126:45447 requested a full tablet report, sending...
I20260810 15:20:37.323562 6319 ts_manager.cc:194] Registered new tserver with Master: 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899)
I20260810 15:20:37.324105 6319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:48303
I20260810 15:20:37.329653 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.323978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.4.64.67:38563)
0810 15:20:37.324147 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.324176 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.324241 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.324365 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.324366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.324579 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:37.324647 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.324649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.325426 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.325427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.326040 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.326042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.326410 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.326411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.326490 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.326889 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:37.326893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:37.327177 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:37.328507 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.328508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.328509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.328686 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.328919 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.328920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.328920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.328989 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.329257 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:37.329259 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:37.329308 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.329401 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.329515 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260810 15:20:37.329643 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.324043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:38563 (local address 127.4.64.126:45447)
0810 15:20:37.324087 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.324089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.324225 (+ 136us) server_negotiation.cc:407] Connection header received
0810 15:20:37.324300 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.324300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.324311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.324360 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:37.324751 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.325333 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.326122 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.326333 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.326412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.327271 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:37.327273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:37.327273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:37.327281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:37.328419 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.328763 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.328764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.328765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.328824 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.329056 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.329057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.329058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.329180 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:37.329255 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.329429 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.329512 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:37.333056 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:37.359167 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.351810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.4.64.126:45447)
0810 15:20:37.351886 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.351887 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.351899 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:37.351936 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.351937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.351947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.352001 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:37.352567 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.353143 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.353961 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.354147 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.354202 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.356568 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:37.356570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:37.356570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:37.356578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:37.357979 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.358367 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.358368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.358369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.358431 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.358639 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.358640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.358640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.358763 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:37.358874 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.358969 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.359057 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:20:37.360986 6319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:37.362767 6319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:37.370327 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.365252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43899 (local address 127.0.0.1:57632)
0810 15:20:37.365609 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.365668 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.365736 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.366031 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.366034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.366037 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.366111 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.366113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.367460 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.367461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.368948 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.368952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.369933 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.369934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.370036 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.370051 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.370168 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.370236 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":62,"thread_start_us":53,"threads_started":1}
I20260810 15:20:37.370656 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.365459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:39655 (local address 127.0.0.1:47548)
0810 15:20:37.365968 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.365976 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.366020 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.366275 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.366275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.366277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.366316 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.366318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.368516 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.368517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.369418 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.369421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.370404 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.370405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.370470 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.370474 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.370533 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.370571 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":81,"threads_started":1}
I20260810 15:20:37.370733 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.365587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.4.64.66:39655)
0810 15:20:37.365864 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.365869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.366074 (+ 205us) server_negotiation.cc:407] Connection header received
0810 15:20:37.366122 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.366124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.366141 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.366209 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.367543 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.368421 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.369504 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.370227 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.370313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.370366 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.370575 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.370622 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":157,"threads_started":1}
I20260810 15:20:37.370880 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.365456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.4.64.67:43899)
0810 15:20:37.365758 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.365763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.365779 (+ 16us) server_negotiation.cc:407] Connection header received
0810 15:20:37.365855 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.365858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.365876 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.365959 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.366204 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.367376 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.369045 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.369830 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.369921 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.369991 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.370575 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.370622 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":80,"threads_started":1}
I20260810 15:20:37.371367 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.365252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45955 (local address 127.0.0.1:35512)
0810 15:20:37.365846 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.365855 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.365894 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.366250 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.366251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.366253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.366317 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.366318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.367527 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.367528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.368948 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.368952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.371066 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.371067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.371133 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.371138 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.371228 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.371298 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":75,"thread_start_us":93,"threads_started":1}
I20260810 15:20:37.371510 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 81cdd7cda35a48d7811210d65dda61e0 (DEFAULT_TABLE table=test-table [id=1527423a4642403aa73d8992a6aff208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:37.371628 6781 tablet_service.cc:1467] Processing CreateTablet for tablet 81cdd7cda35a48d7811210d65dda61e0 (DEFAULT_TABLE table=test-table [id=1527423a4642403aa73d8992a6aff208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:37.371932 6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cdd7cda35a48d7811210d65dda61e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.371933 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cdd7cda35a48d7811210d65dda61e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.374650 6886 tablet_bootstrap.cc:492] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Bootstrap starting.
I20260810 15:20:37.374817 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.365397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.4.64.65:45955)
0810 15:20:37.366097 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.366101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.366111 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:37.366147 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.366149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.366165 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.366242 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.366392 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.367439 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.369767 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.370994 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.371065 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.371119 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.374647 (+ 3528us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.374710 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":101,"threads_started":1}
I20260810 15:20:37.375471 6469 tablet_service.cc:1467] Processing CreateTablet for tablet 81cdd7cda35a48d7811210d65dda61e0 (DEFAULT_TABLE table=test-table [id=1527423a4642403aa73d8992a6aff208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:37.375705 6886 tablet_bootstrap.cc:654] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.375825 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81cdd7cda35a48d7811210d65dda61e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.376073 6886 log.cc:824] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:37.376432 6887 tablet_bootstrap.cc:492] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Bootstrap starting.
I20260810 15:20:37.376859 6886 tablet_bootstrap.cc:492] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: No bootstrap required, opened a new log
I20260810 15:20:37.376916 6886 ts_tablet_manager.cc:1397] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:20:37.378373 6887 tablet_bootstrap.cc:654] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.378630 6886 raft_consensus.cc:348] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.378786 6886 raft_consensus.cc:374] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.378777 6887 log.cc:824] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:37.378829 6886 raft_consensus.cc:729] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.378975 6886 consensus_queue.cc:260] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.379289 6886 ts_tablet_manager.cc:1428] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 15:20:37.379472 6857 heartbeater.cc:503] Master 127.4.64.126:45447 was elected leader, sending a full tablet report...
I20260810 15:20:37.379772 6887 tablet_bootstrap.cc:492] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: No bootstrap required, opened a new log
I20260810 15:20:37.379848 6887 ts_tablet_manager.cc:1397] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:37.381211 6887 raft_consensus.cc:348] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.381393 6887 raft_consensus.cc:374] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.381439 6887 raft_consensus.cc:729] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5711d0ebbd344ff80cfd64e8e6b0078, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.381557 6887 consensus_queue.cc:260] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.381873 6887 ts_tablet_manager.cc:1428] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:20:37.381930 6700 heartbeater.cc:503] Master 127.4.64.126:45447 was elected leader, sending a full tablet report...
I20260810 15:20:37.382141 6891 tablet_bootstrap.cc:492] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Bootstrap starting.
I20260810 15:20:37.382828 6891 tablet_bootstrap.cc:654] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.383137 6891 log.cc:824] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:37.383894 6891 tablet_bootstrap.cc:492] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: No bootstrap required, opened a new log
I20260810 15:20:37.383975 6891 ts_tablet_manager.cc:1397] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:37.385516 6891 raft_consensus.cc:348] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.385718 6891 raft_consensus.cc:374] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.385772 6891 raft_consensus.cc:729] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.385921 6891 consensus_queue.cc:260] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.386186 6891 ts_tablet_manager.cc:1428] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:20:37.386277 6544 heartbeater.cc:503] Master 127.4.64.126:45447 was elected leader, sending a full tablet report...
W20260810 15:20:37.453725 6545 tablet.cc:2369] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:37.492008 6895 raft_consensus.cc:482] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:37.492233 6895 raft_consensus.cc:504] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.492671 6895 leader_election.cc:288] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.495985 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.493103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:39655 (local address 127.4.64.65:36905)
0810 15:20:37.493279 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.493291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.493399 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.493568 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.493569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.493571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.493668 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.493670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.494313 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.494314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.495120 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.495122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.495685 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.495686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.495763 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.495769 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.495842 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.495886 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260810 15:20:37.495966 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.493219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:36905 (local address 127.4.64.66:39655)
0810 15:20:37.493263 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.493265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.493380 (+ 115us) server_negotiation.cc:407] Connection header received
0810 15:20:37.493469 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.493470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.493482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.493548 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.493743 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.494219 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.495221 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.495596 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.495684 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.495695 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.495856 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.495896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:37.496194 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.493019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:47239 (local address 127.4.64.67:43899)
0810 15:20:37.493094 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.493096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.493158 (+ 62us) server_negotiation.cc:407] Connection header received
0810 15:20:37.493261 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.493262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.493274 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.493348 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.493529 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.494135 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.495269 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.495753 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.495854 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.495865 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.496047 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.496097 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:20:37.496306 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cdd7cda35a48d7811210d65dda61e0" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" is_pre_election: true
I20260810 15:20:37.496433 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.492869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43899 (local address 127.4.64.65:47239)
0810 15:20:37.493064 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.493081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.493182 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.493354 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.493355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.493357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.493439 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.493443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.494304 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.494305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.495165 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.495167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.495859 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.495860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.495946 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.495950 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.496201 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.496338 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
I20260810 15:20:37.496476 6649 raft_consensus.cc:2454] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 0.
I20260810 15:20:37.496716 6419 leader_election.cc:302] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c657f2357768432c9a5f3258ebfe1108, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:37.496758 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cdd7cda35a48d7811210d65dda61e0" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" is_pre_election: true
I20260810 15:20:37.496850 6895 raft_consensus.cc:2781] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:37.496923 6804 raft_consensus.cc:2454] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 0.
I20260810 15:20:37.496958 6895 raft_consensus.cc:482] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:37.497002 6895 raft_consensus.cc:3037] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.497901 6895 raft_consensus.cc:504] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.498061 6895 leader_election.cc:288] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 election: Requested vote from peers 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.498302 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cdd7cda35a48d7811210d65dda61e0" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc"
I20260810 15:20:37.498319 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81cdd7cda35a48d7811210d65dda61e0" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
I20260810 15:20:37.498392 6804 raft_consensus.cc:3037] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.498420 6649 raft_consensus.cc:3037] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.499176 6804 raft_consensus.cc:2454] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 1.
I20260810 15:20:37.499186 6649 raft_consensus.cc:2454] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 1.
I20260810 15:20:37.499356 6419 leader_election.cc:302] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c657f2357768432c9a5f3258ebfe1108, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:37.499497 6895 raft_consensus.cc:2781] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:37.499645 6895 raft_consensus.cc:686] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 1 LEADER]: Becoming Leader. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Running, Role: LEADER
I20260810 15:20:37.499753 6895 consensus_queue.cc:237] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } }
I20260810 15:20:37.500762 6319 catalog_manager.cc:5495] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 reported cstate change: term changed from 0 to 1, leader changed from <none> to c657f2357768432c9a5f3258ebfe1108 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "c657f2357768432c9a5f3258ebfe1108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } health_report { overall_health: UNKNOWN } } }
Aug 10 15:20:37 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375236, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:37.549600 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.542411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45447 (local address 127.0.0.1:57074)
0810 15:20:37.542622 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.542646 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.542739 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.542908 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.542908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.543257 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:37.543311 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.543313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.544030 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.544032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.544565 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.544566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.544940 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.544941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.544998 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.545541 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:37.545546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:37.547001 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:37.548421 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.548431 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.548434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.548648 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.548859 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:37.548860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:37.548860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:37.548922 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:37.549182 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:37.549190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:37.549237 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.549335 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.549470 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":112,"threads_started":1}
I20260810 15:20:37.549603 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.542493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.4.64.126:45447)
0810 15:20:37.542601 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.542604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.542718 (+ 114us) server_negotiation.cc:407] Connection header received
0810 15:20:37.542816 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.542816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.542829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.542890 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:37.543419 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.543932 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.544653 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.544862 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.544942 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.547101 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:37.547102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:37.547102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:37.547110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:37.548322 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.548725 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.548726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.548727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.548789 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:37.548992 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:37.548993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:37.548993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:37.549113 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:37.549191 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.549388 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.549526 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":71}
I20260810 15:20:37.551867 6319 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57074:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 15:20:37.552109 6319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:37.553727 6626 tablet_service.cc:1467] Processing CreateTablet for tablet e43cb7de574546bf89f6a0229463380c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b8513cee3c24c53b60de9095e57c01b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:20:37.553697 6469 tablet_service.cc:1467] Processing CreateTablet for tablet e43cb7de574546bf89f6a0229463380c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b8513cee3c24c53b60de9095e57c01b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:20:37.553761 6781 tablet_service.cc:1467] Processing CreateTablet for tablet e43cb7de574546bf89f6a0229463380c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b8513cee3c24c53b60de9095e57c01b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:20:37.553905 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43cb7de574546bf89f6a0229463380c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.553925 6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43cb7de574546bf89f6a0229463380c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.553926 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43cb7de574546bf89f6a0229463380c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.555106 6887 tablet_bootstrap.cc:492] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: Bootstrap starting.
I20260810 15:20:37.555511 6891 tablet_bootstrap.cc:492] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: Bootstrap starting.
I20260810 15:20:37.555914 6887 tablet_bootstrap.cc:654] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.555972 6891 tablet_bootstrap.cc:654] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.556365 6886 tablet_bootstrap.cc:492] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Bootstrap starting.
I20260810 15:20:37.556764 6891 tablet_bootstrap.cc:492] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: No bootstrap required, opened a new log
I20260810 15:20:37.556825 6891 ts_tablet_manager.cc:1397] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:37.556913 6887 tablet_bootstrap.cc:492] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: No bootstrap required, opened a new log
I20260810 15:20:37.556968 6887 ts_tablet_manager.cc:1397] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:37.557052 6886 tablet_bootstrap.cc:654] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.557147 6887 raft_consensus.cc:348] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.557241 6887 raft_consensus.cc:374] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.557276 6887 raft_consensus.cc:729] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5711d0ebbd344ff80cfd64e8e6b0078, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.557217 6891 raft_consensus.cc:348] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.557307 6891 raft_consensus.cc:374] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.557332 6887 consensus_queue.cc:260] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.557354 6891 raft_consensus.cc:729] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.557433 6887 ts_tablet_manager.cc:1428] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:37.557417 6891 consensus_queue.cc:260] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.557514 6891 ts_tablet_manager.cc:1428] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:37.557709 6893 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.557832 6893 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:37.557886 6886 tablet_bootstrap.cc:492] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: No bootstrap required, opened a new log
I20260810 15:20:37.557930 6886 ts_tablet_manager.cc:1397] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:20:37.557997 6895 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.558094 6895 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:37.558091 6886 raft_consensus.cc:348] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.558172 6886 raft_consensus.cc:374] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.558199 6886 raft_consensus.cc:729] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.558247 6886 consensus_queue.cc:260] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.558365 6886 ts_tablet_manager.cc:1428] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:37.558440 6892 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.558557 6892 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 15:20:37.569921 6858 tablet.cc:2369] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:20:37.631645 6701 tablet.cc:2369] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:37.635083 6892 raft_consensus.cc:482] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:37.635179 6892 raft_consensus.cc:504] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.635670 6892 leader_election.cc:288] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.639062 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.635942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:49811 (local address 127.4.64.66:39655)
0810 15:20:37.635990 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.635993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.636006 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:20:37.636040 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.636041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.636052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.636115 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.636284 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.636799 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.638079 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.638626 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.638730 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.638744 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.638928 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.638995 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:37.639062 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.635874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:39655 (local address 127.4.64.67:49811)
0810 15:20:37.635914 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.635928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.635978 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.636128 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.636129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.636132 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.636203 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.636205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.636910 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.636911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.637963 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.637966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.638732 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.638733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.638839 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.638843 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.638916 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.638970 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 15:20:37.639268 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.636038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:42749 (local address 127.4.64.65:45955)
0810 15:20:37.636082 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.636084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.636196 (+ 112us) server_negotiation.cc:407] Connection header received
0810 15:20:37.636263 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.636263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.636276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.636326 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:37.636469 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.637006 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.638487 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.638875 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.638968 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.638981 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.639161 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.639202 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:37.639292 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.635922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45955 (local address 127.4.64.67:42749)
0810 15:20:37.636124 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.636138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.636209 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.636327 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.636328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.636329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:37.636404 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.636405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.637111 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.637112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.638382 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.638385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.638980 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.638981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.639071 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.639074 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.639155 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.639195 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":139,"threads_started":1}
I20260810 15:20:37.639386 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43cb7de574546bf89f6a0229463380c" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" is_pre_election: true
I20260810 15:20:37.639494 6649 raft_consensus.cc:2454] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 0.
I20260810 15:20:37.639632 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43cb7de574546bf89f6a0229463380c" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c657f2357768432c9a5f3258ebfe1108" is_pre_election: true
I20260810 15:20:37.639662 6729 leader_election.cc:302] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbb6c37f8bd4e03ac127185d1ffa5bc, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:37.639727 6496 raft_consensus.cc:2454] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 0.
I20260810 15:20:37.639801 6892 raft_consensus.cc:2781] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:37.639851 6892 raft_consensus.cc:482] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:37.639902 6892 raft_consensus.cc:3037] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.640594 6892 raft_consensus.cc:504] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.640745 6892 leader_election.cc:288] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 election: Requested vote from peers c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.640972 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43cb7de574546bf89f6a0229463380c" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
I20260810 15:20:37.641053 6649 raft_consensus.cc:3037] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.641709 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43cb7de574546bf89f6a0229463380c" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c657f2357768432c9a5f3258ebfe1108"
I20260810 15:20:37.641789 6496 raft_consensus.cc:3037] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.641805 6649 raft_consensus.cc:2454] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 1.
I20260810 15:20:37.641950 6729 leader_election.cc:302] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbb6c37f8bd4e03ac127185d1ffa5bc, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:37.642053 6892 raft_consensus.cc:2781] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:37.642210 6892 raft_consensus.cc:686] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 LEADER]: Becoming Leader. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Running, Role: LEADER
I20260810 15:20:37.642314 6892 consensus_queue.cc:237] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.642758 6496 raft_consensus.cc:2454] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 1.
I20260810 15:20:37.642936 6911 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: TxnStatusTablet state changed. Reason: New leader 9fbb6c37f8bd4e03ac127185d1ffa5bc. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.643041 6911 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:20:37.643229 6913 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 15:20:37.643203 6319 catalog_manager.cc:5495] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67). New cstate: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } health_report { overall_health: HEALTHY } } }
I20260810 15:20:37.643352 6913 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 15:20:37.654760 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.651738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43899 (local address 127.0.0.1:57646)
0810 15:20:37.651820 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.651833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.651907 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.652083 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.652084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.652085 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:37.652134 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.652137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.653024 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.653025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.653779 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.653784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.654259 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.654260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.654313 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.654330 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.654615 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.654622 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.654650 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.654697 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:20:37.654778 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.651820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.4.64.67:43899)
0810 15:20:37.651870 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.651872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.651887 (+ 15us) server_negotiation.cc:407] Connection header received
0810 15:20:37.651979 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.651980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.652004 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.652083 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:37.652174 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.652940 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.653863 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.654177 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.654255 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.654422 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.654550 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.654632 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.654657 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.654687 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:20:37.656695 6496 raft_consensus.cc:1261] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:37.656708 6649 raft_consensus.cc:1261] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:37.657092 6892 consensus_queue.cc:1035] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:37.657233 6911 consensus_queue.cc:1035] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:37.658407 6895 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: TxnStatusTablet state changed. Reason: New leader 9fbb6c37f8bd4e03ac127185d1ffa5bc. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.658444 6893 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: TxnStatusTablet state changed. Reason: New leader 9fbb6c37f8bd4e03ac127185d1ffa5bc. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.658535 6895 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:37.658535 6893 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:37.659720 6892 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.659813 6892 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:20:37.659991 6911 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.660066 6911 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P 9fbb6c37f8bd4e03ac127185d1ffa5bc: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:20:37.660471 6895 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.660557 6895 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P c657f2357768432c9a5f3258ebfe1108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:37.660564 6893 tablet_replica.cc:439] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } } }
I20260810 15:20:37.660651 6893 tablet_replica.cc:442] T e43cb7de574546bf89f6a0229463380c P d5711d0ebbd344ff80cfd64e8e6b0078: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:20:37.661123 6916 mvcc.cc:205] Tried to move back new op lower bound from 7316992973439107072 to 7316992973383446528. Current Snapshot: MvccSnapshot[applied={T|T < 7316992973439107072}]
I20260810 15:20:37.665964 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.663071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.4.64.65:45955)
0810 15:20:37.663111 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.663113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.663122 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:20:37.663179 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.663180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.663199 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.663270 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:37.663435 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.664195 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.665000 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.665324 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.665397 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.665536 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.665734 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.665845 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.665868 (+ 23us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.665892 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:37.671298 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.668551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:51893 (local address 127.4.64.67:43899)
0810 15:20:37.668623 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.668625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.668736 (+ 111us) server_negotiation.cc:407] Connection header received
0810 15:20:37.668774 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.668774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.668785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.668852 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:37.668944 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.669688 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.670436 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.670741 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.670819 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.670994 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.671028 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.671081 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.671191 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.671213 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:20:37.671324 6930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.668494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43899 (local address 127.4.64.65:51893)
0810 15:20:37.668659 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.668670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.668730 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.668855 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.668855 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.668856 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:37.668908 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.668908 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.669756 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.669757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.670350 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.670351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.670826 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.670826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.670886 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.670900 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.671110 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.671113 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.671179 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.671228 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260810 15:20:37.673749 6649 raft_consensus.cc:1261] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Refusing update from remote peer c657f2357768432c9a5f3258ebfe1108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:37.673833 6804 raft_consensus.cc:1261] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Refusing update from remote peer c657f2357768432c9a5f3258ebfe1108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:37.674028 6895 consensus_queue.cc:1035] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:37.674217 6898 consensus_queue.cc:1035] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:37.676054 6917 mvcc.cc:205] Tried to move back new op lower bound from 7316992973509955584 to 7316992972800065536. Current Snapshot: MvccSnapshot[applied={T|T < 7316992973509955584}]
I20260810 15:20:37.682837 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.680181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45955 (local address 127.4.64.67:51409)
0810 15:20:37.680254 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.680266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.680312 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.680559 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.680560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.680561 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:37.680621 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.680622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.681265 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.681266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.681941 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.681943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.682400 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.682401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.682470 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.682492 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.682672 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.682674 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.682725 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.682769 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 15:20:37.682821 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.680376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:51409 (local address 127.4.64.65:45955)
0810 15:20:37.680437 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.680438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.680445 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:37.680465 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.680466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.680476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.680543 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:37.680704 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.681174 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.682027 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.682306 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.682398 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.682575 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.682612 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.682665 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.682730 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.682749 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:20:37.692488 6320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:37.692612 6320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:37.693434 6320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Sending DeleteTablet for 3 replicas of tablet 81cdd7cda35a48d7811210d65dda61e0
I20260810 15:20:37.693775 6469 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cdd7cda35a48d7811210d65dda61e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:35512
I20260810 15:20:37.693825 6626 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cdd7cda35a48d7811210d65dda61e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47548
I20260810 15:20:37.694097 6781 tablet_service.cc:1514] Processing DeleteTablet for tablet 81cdd7cda35a48d7811210d65dda61e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57632
I20260810 15:20:37.694106 6320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:37.694248 6320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:37.694309 6941 tablet_replica.cc:330] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: stopping tablet replica
I20260810 15:20:37.694432 6941 raft_consensus.cc:2229] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:37.694499 6941 raft_consensus.cc:2258] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:37.694948 6939 tablet_replica.cc:330] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: stopping tablet replica
I20260810 15:20:37.695524 6469 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcefd145d834632a470756c75ad2c97 (DEFAULT_TABLE table=test-table [id=ff98cdae234242e9ae107e66461060c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:37.695642 6939 raft_consensus.cc:2229] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:37.695626 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcefd145d834632a470756c75ad2c97 (DEFAULT_TABLE table=test-table [id=ff98cdae234242e9ae107e66461060c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:37.695667 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcefd145d834632a470756c75ad2c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.695761 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcefd145d834632a470756c75ad2c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.695909 6939 raft_consensus.cc:2258] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:37.696415 6939 ts_tablet_manager.cc:1905] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:37.697147 6891 tablet_bootstrap.cc:492] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Bootstrap starting.
I20260810 15:20:37.697285 6781 tablet_service.cc:1467] Processing CreateTablet for tablet 1fcefd145d834632a470756c75ad2c97 (DEFAULT_TABLE table=test-table [id=ff98cdae234242e9ae107e66461060c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:37.697422 6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fcefd145d834632a470756c75ad2c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:37.697579 6891 tablet_bootstrap.cc:654] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.698033 6887 tablet_bootstrap.cc:492] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Bootstrap starting.
I20260810 15:20:37.698200 6940 tablet_replica.cc:330] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: stopping tablet replica
I20260810 15:20:37.698377 6940 raft_consensus.cc:2229] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:37.698442 6887 tablet_bootstrap.cc:654] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.698546 6940 raft_consensus.cc:2258] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:37.699184 6886 tablet_bootstrap.cc:492] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Bootstrap starting.
I20260810 15:20:37.699626 6940 ts_tablet_manager.cc:1905] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:37.699668 6886 tablet_bootstrap.cc:654] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:37.700305 6939 ts_tablet_manager.cc:1918] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:37.700413 6939 log.cc:1196] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/81cdd7cda35a48d7811210d65dda61e0
I20260810 15:20:37.700690 6939 ts_tablet_manager.cc:1939] T 81cdd7cda35a48d7811210d65dda61e0 P c657f2357768432c9a5f3258ebfe1108: Deleting consensus metadata
I20260810 15:20:37.699189 6941 ts_tablet_manager.cc:1905] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:37.701889 6302 catalog_manager.cc:4841] TS c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955): tablet 81cdd7cda35a48d7811210d65dda61e0 (table test-table [id=1527423a4642403aa73d8992a6aff208]) successfully deleted
I20260810 15:20:37.702734 6940 ts_tablet_manager.cc:1918] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:37.702790 6886 tablet_bootstrap.cc:492] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: No bootstrap required, opened a new log
I20260810 15:20:37.702812 6940 log.cc:1196] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/81cdd7cda35a48d7811210d65dda61e0
I20260810 15:20:37.702831 6886 ts_tablet_manager.cc:1397] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260810 15:20:37.702996 6886 raft_consensus.cc:348] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.703104 6940 ts_tablet_manager.cc:1939] T 81cdd7cda35a48d7811210d65dda61e0 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting consensus metadata
I20260810 15:20:37.703112 6886 raft_consensus.cc:374] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.703138 6886 raft_consensus.cc:729] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.703197 6886 consensus_queue.cc:260] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.703380 6886 ts_tablet_manager.cc:1428] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:37.703614 6305 catalog_manager.cc:4841] TS d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655): tablet 81cdd7cda35a48d7811210d65dda61e0 (table test-table [id=1527423a4642403aa73d8992a6aff208]) successfully deleted
I20260810 15:20:37.704139 6941 ts_tablet_manager.cc:1918] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:37.704213 6941 log.cc:1196] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/81cdd7cda35a48d7811210d65dda61e0
I20260810 15:20:37.704473 6941 ts_tablet_manager.cc:1939] T 81cdd7cda35a48d7811210d65dda61e0 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting consensus metadata
I20260810 15:20:37.705049 6891 tablet_bootstrap.cc:492] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: No bootstrap required, opened a new log
I20260810 15:20:37.705111 6891 ts_tablet_manager.cc:1397] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260810 15:20:37.705154 6887 tablet_bootstrap.cc:492] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: No bootstrap required, opened a new log
I20260810 15:20:37.705199 6887 ts_tablet_manager.cc:1397] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 15:20:37.705269 6891 raft_consensus.cc:348] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.705322 6891 raft_consensus.cc:374] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.705335 6891 raft_consensus.cc:729] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.705334 6887 raft_consensus.cc:348] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.705399 6887 raft_consensus.cc:374] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:37.705420 6887 raft_consensus.cc:729] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5711d0ebbd344ff80cfd64e8e6b0078, State: Initialized, Role: FOLLOWER
I20260810 15:20:37.705397 6891 consensus_queue.cc:260] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.705467 6887 consensus_queue.cc:260] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.705511 6891 ts_tablet_manager.cc:1428] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.705555 6887 ts_tablet_manager.cc:1428] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:37.705824 6305 catalog_manager.cc:4841] TS 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899): tablet 81cdd7cda35a48d7811210d65dda61e0 (table test-table [id=1527423a4642403aa73d8992a6aff208]) successfully deleted
I20260810 15:20:37.843863 6931 raft_consensus.cc:482] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:37.843991 6931 raft_consensus.cc:504] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.844240 6931 leader_election.cc:288] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.844461 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcefd145d834632a470756c75ad2c97" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c657f2357768432c9a5f3258ebfe1108" is_pre_election: true
I20260810 15:20:37.844483 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcefd145d834632a470756c75ad2c97" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" is_pre_election: true
I20260810 15:20:37.844575 6496 raft_consensus.cc:2454] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 0.
I20260810 15:20:37.844604 6649 raft_consensus.cc:2454] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 0.
I20260810 15:20:37.844817 6726 leader_election.cc:302] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbb6c37f8bd4e03ac127185d1ffa5bc, c657f2357768432c9a5f3258ebfe1108; no voters:
I20260810 15:20:37.845000 6931 raft_consensus.cc:2781] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:37.845067 6931 raft_consensus.cc:482] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:37.845098 6931 raft_consensus.cc:3037] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.846113 6931 raft_consensus.cc:504] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.846297 6931 leader_election.cc:288] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 election: Requested vote from peers c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:37.846439 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcefd145d834632a470756c75ad2c97" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c657f2357768432c9a5f3258ebfe1108"
I20260810 15:20:37.846518 6496 raft_consensus.cc:3037] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.846545 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fcefd145d834632a470756c75ad2c97" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
I20260810 15:20:37.846628 6649 raft_consensus.cc:3037] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:37.847244 6649 raft_consensus.cc:2454] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 1.
I20260810 15:20:37.847308 6496 raft_consensus.cc:2454] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 1.
I20260810 15:20:37.847476 6726 leader_election.cc:302] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbb6c37f8bd4e03ac127185d1ffa5bc, c657f2357768432c9a5f3258ebfe1108; no voters:
I20260810 15:20:37.847636 6931 raft_consensus.cc:2781] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:37.847729 6931 raft_consensus.cc:686] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 LEADER]: Becoming Leader. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Running, Role: LEADER
I20260810 15:20:37.847788 6931 consensus_queue.cc:237] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:37.848579 6319 catalog_manager.cc:5495] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67). New cstate: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } health_report { overall_health: HEALTHY } } }
I20260810 15:20:37.884014 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.881599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.4.64.67:43899)
0810 15:20:37.881684 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.881686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.881696 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:37.881768 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.881769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.881780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.881836 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:37.881993 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.882492 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.883240 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.883492 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.883571 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.883733 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.883774 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.883832 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.883897 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.883917 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 15:20:37.888567 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.886478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:42249 (local address 127.4.64.67:43899)
0810 15:20:37.886511 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:37.886513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:37.886522 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:20:37.886552 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:37.886553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:37.886562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:37.886605 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:37.886703 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.887158 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.887903 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.888105 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.888180 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.888317 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.888349 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.888392 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:37.888447 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:20:37.888471 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:37.888567 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:37.886394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43899 (local address 127.4.64.67:42249)
0810 15:20:37.886449 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:37.886461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:37.886501 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:37.886607 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:37.886607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:37.886608 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:20:37.886650 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.886651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.887246 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.887247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.887782 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:37.887783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:37.888177 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:37.888177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:37.888230 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:37.888238 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:20:37.888402 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:20:37.888403 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:20:37.888447 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:20:37.888498 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":6}
I20260810 15:20:37.890816 6649 raft_consensus.cc:1261] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:37.890844 6496 raft_consensus.cc:1261] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:37.891079 6911 consensus_queue.cc:1035] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:37.891299 6911 consensus_queue.cc:1035] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:37.892910 6918 mvcc.cc:205] Tried to move back new op lower bound from 7316992974399553536 to 7316992974225100800. Current Snapshot: MvccSnapshot[applied={T|T < 7316992974399553536}]
W20260810 15:20:37.896481 6916 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 15:20:39.416862 6318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:39.416980 6318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:39.418021 6318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Sending DeleteTablet for 3 replicas of tablet 1fcefd145d834632a470756c75ad2c97
I20260810 15:20:39.418295 6469 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcefd145d834632a470756c75ad2c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:35512
I20260810 15:20:39.418344 6781 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcefd145d834632a470756c75ad2c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57632
I20260810 15:20:39.418503 6626 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fcefd145d834632a470756c75ad2c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47548
I20260810 15:20:39.418706 6975 tablet_replica.cc:330] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: stopping tablet replica
I20260810 15:20:39.418713 6973 tablet_replica.cc:330] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: stopping tablet replica
I20260810 15:20:39.418772 6973 raft_consensus.cc:2229] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:39.418799 6975 raft_consensus.cc:2229] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:39.418824 6973 raft_consensus.cc:2258] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:39.418892 6975 raft_consensus.cc:2258] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:39.418958 6974 tablet_replica.cc:330] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: stopping tablet replica
I20260810 15:20:39.419020 6974 raft_consensus.cc:2229] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:39.419128 6974 raft_consensus.cc:2258] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:39.419391 6973 ts_tablet_manager.cc:1905] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:39.419390 6974 ts_tablet_manager.cc:1905] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:39.421444 6975 ts_tablet_manager.cc:1905] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:39.421459 6973 ts_tablet_manager.cc:1918] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:20:39.421519 6973 log.cc:1196] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/1fcefd145d834632a470756c75ad2c97
I20260810 15:20:39.421811 6973 ts_tablet_manager.cc:1939] T 1fcefd145d834632a470756c75ad2c97 P c657f2357768432c9a5f3258ebfe1108: Deleting consensus metadata
I20260810 15:20:39.422169 6974 ts_tablet_manager.cc:1918] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:20:39.422199 6302 catalog_manager.cc:4841] TS c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955): tablet 1fcefd145d834632a470756c75ad2c97 (table test-table [id=ff98cdae234242e9ae107e66461060c2]) successfully deleted
I20260810 15:20:39.422238 6974 log.cc:1196] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/1fcefd145d834632a470756c75ad2c97
I20260810 15:20:39.422533 6974 ts_tablet_manager.cc:1939] T 1fcefd145d834632a470756c75ad2c97 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting consensus metadata
I20260810 15:20:39.422907 6305 catalog_manager.cc:4841] TS 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899): tablet 1fcefd145d834632a470756c75ad2c97 (table test-table [id=ff98cdae234242e9ae107e66461060c2]) successfully deleted
I20260810 15:20:39.423647 6975 ts_tablet_manager.cc:1918] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:20:39.423718 6975 log.cc:1196] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/1fcefd145d834632a470756c75ad2c97
I20260810 15:20:39.423925 6975 ts_tablet_manager.cc:1939] T 1fcefd145d834632a470756c75ad2c97 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting consensus metadata
I20260810 15:20:39.424582 6305 catalog_manager.cc:4841] TS d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655): tablet 1fcefd145d834632a470756c75ad2c97 (table test-table [id=ff98cdae234242e9ae107e66461060c2]) successfully deleted
Aug 10 15:20:39 dist-test-slave-9zdj krb5kdc[6270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375239, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:20:39 dist-test-slave-9zdj krb5kdc[6270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375239, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:39.447310 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:39.438869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.4.64.126:45447)
0810 15:20:39.439141 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:39.439143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:39.439154 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:39.439181 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:39.439182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:39.439194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:39.439274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:39.439944 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:39.440588 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:39.441424 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:39.441815 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:39.441882 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:39.444476 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:39.444478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:39.444478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:39.444486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:39.446144 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:39.446566 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:39.446567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:39.446567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:39.446628 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:39.446818 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:39.446819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:39.446819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:39.446940 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:39.447022 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:39.447111 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 15:20:39.447180 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":157,"threads_started":1}
I20260810 15:20:39.449100 6318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:39.449290 6318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:39.450995 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 49ac003ee3b5435ba3427d5ec97ce499 (DEFAULT_TABLE table=test-table [id=05b402f852964aae8c46e1bcc1a72e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:39.451031 6781 tablet_service.cc:1467] Processing CreateTablet for tablet 49ac003ee3b5435ba3427d5ec97ce499 (DEFAULT_TABLE table=test-table [id=05b402f852964aae8c46e1bcc1a72e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:39.451154 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ac003ee3b5435ba3427d5ec97ce499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:39.451231 6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ac003ee3b5435ba3427d5ec97ce499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:39.452521 6469 tablet_service.cc:1467] Processing CreateTablet for tablet 49ac003ee3b5435ba3427d5ec97ce499 (DEFAULT_TABLE table=test-table [id=05b402f852964aae8c46e1bcc1a72e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:39.452543 6989 tablet_bootstrap.cc:492] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Bootstrap starting.
I20260810 15:20:39.452677 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ac003ee3b5435ba3427d5ec97ce499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:39.452740 6990 tablet_bootstrap.cc:492] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Bootstrap starting.
I20260810 15:20:39.453154 6989 tablet_bootstrap.cc:654] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:39.453156 6990 tablet_bootstrap.cc:654] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:39.453943 6990 tablet_bootstrap.cc:492] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: No bootstrap required, opened a new log
I20260810 15:20:39.454005 6990 ts_tablet_manager.cc:1397] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:39.454131 6990 raft_consensus.cc:348] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.454221 6990 raft_consensus.cc:374] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:39.454272 6990 raft_consensus.cc:729] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Initialized, Role: FOLLOWER
I20260810 15:20:39.454324 6990 consensus_queue.cc:260] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.454396 6993 tablet_bootstrap.cc:492] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Bootstrap starting.
I20260810 15:20:39.454425 6990 ts_tablet_manager.cc:1428] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:39.454746 6989 tablet_bootstrap.cc:492] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: No bootstrap required, opened a new log
I20260810 15:20:39.454825 6989 ts_tablet_manager.cc:1397] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:39.454880 6993 tablet_bootstrap.cc:654] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:39.454962 6989 raft_consensus.cc:348] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.455040 6989 raft_consensus.cc:374] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:39.455070 6989 raft_consensus.cc:729] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5711d0ebbd344ff80cfd64e8e6b0078, State: Initialized, Role: FOLLOWER
I20260810 15:20:39.455120 6989 consensus_queue.cc:260] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.455310 6989 ts_tablet_manager.cc:1428] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:39.455633 6993 tablet_bootstrap.cc:492] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: No bootstrap required, opened a new log
I20260810 15:20:39.455710 6993 ts_tablet_manager.cc:1397] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:39.455837 6993 raft_consensus.cc:348] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.455922 6993 raft_consensus.cc:374] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:39.455955 6993 raft_consensus.cc:729] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Initialized, Role: FOLLOWER
I20260810 15:20:39.456005 6993 consensus_queue.cc:260] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.456207 6993 ts_tablet_manager.cc:1428] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:39.471622 6954 raft_consensus.cc:482] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:39.471737 6954 raft_consensus.cc:504] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.471975 6954 leader_election.cc:288] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:39.472234 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac003ee3b5435ba3427d5ec97ce499" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c657f2357768432c9a5f3258ebfe1108" is_pre_election: true
I20260810 15:20:39.472352 6496 raft_consensus.cc:2454] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 0.
I20260810 15:20:39.472283 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac003ee3b5435ba3427d5ec97ce499" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" is_pre_election: true
I20260810 15:20:39.472432 6649 raft_consensus.cc:2454] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 0.
I20260810 15:20:39.472530 6726 leader_election.cc:302] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbb6c37f8bd4e03ac127185d1ffa5bc, c657f2357768432c9a5f3258ebfe1108; no voters:
I20260810 15:20:39.472644 6954 raft_consensus.cc:2781] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:39.472684 6954 raft_consensus.cc:482] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:39.472716 6954 raft_consensus.cc:3037] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:39.473704 6954 raft_consensus.cc:504] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.473861 6954 leader_election.cc:288] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 election: Requested vote from peers c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955), d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655)
I20260810 15:20:39.474006 6496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac003ee3b5435ba3427d5ec97ce499" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c657f2357768432c9a5f3258ebfe1108"
I20260810 15:20:39.474031 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ac003ee3b5435ba3427d5ec97ce499" candidate_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
I20260810 15:20:39.474108 6496 raft_consensus.cc:3037] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:39.474108 6649 raft_consensus.cc:3037] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:39.474903 6649 raft_consensus.cc:2454] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 1.
I20260810 15:20:39.474903 6496 raft_consensus.cc:2454] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fbb6c37f8bd4e03ac127185d1ffa5bc in term 1.
I20260810 15:20:39.475122 6729 leader_election.cc:302] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fbb6c37f8bd4e03ac127185d1ffa5bc, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:39.475251 6954 raft_consensus.cc:2781] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:39.475306 6954 raft_consensus.cc:686] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 LEADER]: Becoming Leader. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Running, Role: LEADER
I20260810 15:20:39.475365 6954 consensus_queue.cc:237] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.475937 6319 catalog_manager.cc:5495] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67). New cstate: current_term: 1 leader_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } health_report { overall_health: HEALTHY } } }
I20260810 15:20:39.485783 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:39.483401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.4.64.67:43899)
0810 15:20:39.483554 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:39.483556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:39.483580 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:20:39.483636 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:39.483637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:39.483647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:39.483728 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:39.483841 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:39.484341 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:39.485130 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:39.485340 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:39.485398 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:39.485526 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:39.485554 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:39.485588 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:39.485695 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 15:20:39.485715 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260810 15:20:39.488174 6649 raft_consensus.cc:1261] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:39.488281 6496 raft_consensus.cc:1261] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Refusing update from remote peer 9fbb6c37f8bd4e03ac127185d1ffa5bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:39.488404 6954 consensus_queue.cc:1035] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:39.488512 6892 consensus_queue.cc:1035] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:39.499004 6318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:39.499145 6318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:39.499790 6318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Sending DeleteTablet for 3 replicas of tablet 49ac003ee3b5435ba3427d5ec97ce499
I20260810 15:20:39.500053 6626 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ac003ee3b5435ba3427d5ec97ce499 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47548
I20260810 15:20:39.500061 6781 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ac003ee3b5435ba3427d5ec97ce499 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57632
I20260810 15:20:39.500080 6469 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ac003ee3b5435ba3427d5ec97ce499 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:35512
I20260810 15:20:39.500160 6975 tablet_replica.cc:330] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: stopping tablet replica
I20260810 15:20:39.500159 6974 tablet_replica.cc:330] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: stopping tablet replica
I20260810 15:20:39.500191 6973 tablet_replica.cc:330] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: stopping tablet replica
I20260810 15:20:39.500213 6975 raft_consensus.cc:2229] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:39.500221 6974 raft_consensus.cc:2229] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:39.500247 6973 raft_consensus.cc:2229] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:39.500260 6975 raft_consensus.cc:2258] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:39.500275 6974 raft_consensus.cc:2258] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:39.500291 6973 raft_consensus.cc:2258] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:39.500643 6975 ts_tablet_manager.cc:1905] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:39.500782 6974 ts_tablet_manager.cc:1905] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:39.500756 6318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:39.500903 6318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:39.502151 6781 tablet_service.cc:1467] Processing CreateTablet for tablet 2012d9ebc5cc45589ad8cc55ba3ce165 (DEFAULT_TABLE table=test-table [id=b917974890be4a0095459f9c6dc5328b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:39.502281 6781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2012d9ebc5cc45589ad8cc55ba3ce165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:39.502511 6469 tablet_service.cc:1467] Processing CreateTablet for tablet 2012d9ebc5cc45589ad8cc55ba3ce165 (DEFAULT_TABLE table=test-table [id=b917974890be4a0095459f9c6dc5328b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:39.502570 6626 tablet_service.cc:1467] Processing CreateTablet for tablet 2012d9ebc5cc45589ad8cc55ba3ce165 (DEFAULT_TABLE table=test-table [id=b917974890be4a0095459f9c6dc5328b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:39.502601 6974 ts_tablet_manager.cc:1918] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:39.502650 6974 log.cc:1196] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/49ac003ee3b5435ba3427d5ec97ce499
I20260810 15:20:39.502650 6469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2012d9ebc5cc45589ad8cc55ba3ce165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:39.502647 6975 ts_tablet_manager.cc:1918] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:39.502681 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2012d9ebc5cc45589ad8cc55ba3ce165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:39.502698 6975 log.cc:1196] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/49ac003ee3b5435ba3427d5ec97ce499
I20260810 15:20:39.502902 6974 ts_tablet_manager.cc:1939] T 49ac003ee3b5435ba3427d5ec97ce499 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting consensus metadata
I20260810 15:20:39.502928 6975 ts_tablet_manager.cc:1939] T 49ac003ee3b5435ba3427d5ec97ce499 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting consensus metadata
I20260810 15:20:39.503960 6993 tablet_bootstrap.cc:492] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108: Bootstrap starting.
I20260810 15:20:39.504159 6973 ts_tablet_manager.cc:1905] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:39.504241 6990 tablet_bootstrap.cc:492] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Bootstrap starting.
I20260810 15:20:39.504287 6989 tablet_bootstrap.cc:492] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Bootstrap starting.
I20260810 15:20:39.504349 6993 tablet_bootstrap.cc:654] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:39.504551 6990 tablet_bootstrap.cc:654] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:39.504681 6989 tablet_bootstrap.cc:654] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:39.505458 6305 catalog_manager.cc:4841] TS d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655): tablet 49ac003ee3b5435ba3427d5ec97ce499 (table test-table [id=05b402f852964aae8c46e1bcc1a72e64]) successfully deleted
I20260810 15:20:39.505707 6989 tablet_bootstrap.cc:492] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: No bootstrap required, opened a new log
I20260810 15:20:39.505746 6973 ts_tablet_manager.cc:1918] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:20:39.505764 6989 ts_tablet_manager.cc:1397] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:39.505797 6973 log.cc:1196] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/49ac003ee3b5435ba3427d5ec97ce499
I20260810 15:20:39.505923 6989 raft_consensus.cc:348] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.506011 6989 raft_consensus.cc:374] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:39.506028 6973 ts_tablet_manager.cc:1939] T 49ac003ee3b5435ba3427d5ec97ce499 P c657f2357768432c9a5f3258ebfe1108: Deleting consensus metadata
I20260810 15:20:39.506034 6989 raft_consensus.cc:729] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5711d0ebbd344ff80cfd64e8e6b0078, State: Initialized, Role: FOLLOWER
I20260810 15:20:39.506088 6989 consensus_queue.cc:260] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.506191 6993 tablet_bootstrap.cc:492] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108: No bootstrap required, opened a new log
I20260810 15:20:39.506197 6989 ts_tablet_manager.cc:1428] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:39.506239 6993 ts_tablet_manager.cc:1397] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:39.506356 6993 raft_consensus.cc:348] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.505606 6305 catalog_manager.cc:4841] TS 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899): tablet 49ac003ee3b5435ba3427d5ec97ce499 (table test-table [id=05b402f852964aae8c46e1bcc1a72e64]) successfully deleted
I20260810 15:20:39.506423 6993 raft_consensus.cc:374] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:39.506446 6993 raft_consensus.cc:729] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Initialized, Role: FOLLOWER
I20260810 15:20:39.506379 6302 catalog_manager.cc:4841] TS c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955): tablet 49ac003ee3b5435ba3427d5ec97ce499 (table test-table [id=05b402f852964aae8c46e1bcc1a72e64]) successfully deleted
I20260810 15:20:39.506493 6993 consensus_queue.cc:260] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.506574 6990 tablet_bootstrap.cc:492] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: No bootstrap required, opened a new log
I20260810 15:20:39.506590 6993 ts_tablet_manager.cc:1428] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:39.506619 6990 ts_tablet_manager.cc:1397] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:39.506736 6990 raft_consensus.cc:348] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.506803 6990 raft_consensus.cc:374] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:39.506822 6990 raft_consensus.cc:729] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbb6c37f8bd4e03ac127185d1ffa5bc, State: Initialized, Role: FOLLOWER
I20260810 15:20:39.506861 6990 consensus_queue.cc:260] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.506970 6990 ts_tablet_manager.cc:1428] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:39.511879 6996 raft_consensus.cc:482] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:39.511951 6996 raft_consensus.cc:504] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.512099 6996 leader_election.cc:288] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655), 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899)
I20260810 15:20:39.512255 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2012d9ebc5cc45589ad8cc55ba3ce165" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" is_pre_election: true
I20260810 15:20:39.512341 6649 raft_consensus.cc:2454] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 0.
I20260810 15:20:39.512370 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2012d9ebc5cc45589ad8cc55ba3ce165" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" is_pre_election: true
I20260810 15:20:39.512470 6804 raft_consensus.cc:2454] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 0.
I20260810 15:20:39.512485 6419 leader_election.cc:302] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c657f2357768432c9a5f3258ebfe1108, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:39.512570 6996 raft_consensus.cc:2781] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:39.512615 6996 raft_consensus.cc:482] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:39.512650 6996 raft_consensus.cc:3037] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:39.513437 6996 raft_consensus.cc:504] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.513576 6996 leader_election.cc:288] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 election: Requested vote from peers d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655), 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899)
I20260810 15:20:39.513734 6649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2012d9ebc5cc45589ad8cc55ba3ce165" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078"
I20260810 15:20:39.513803 6649 raft_consensus.cc:3037] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:39.513790 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2012d9ebc5cc45589ad8cc55ba3ce165" candidate_uuid: "c657f2357768432c9a5f3258ebfe1108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc"
I20260810 15:20:39.513875 6804 raft_consensus.cc:3037] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:39.514464 6649 raft_consensus.cc:2454] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 1.
I20260810 15:20:39.514531 6804 raft_consensus.cc:2454] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c657f2357768432c9a5f3258ebfe1108 in term 1.
I20260810 15:20:39.514701 6419 leader_election.cc:302] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c657f2357768432c9a5f3258ebfe1108, d5711d0ebbd344ff80cfd64e8e6b0078; no voters:
I20260810 15:20:39.514878 6996 raft_consensus.cc:2781] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:39.515021 6996 raft_consensus.cc:686] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [term 1 LEADER]: Becoming Leader. State: Replica: c657f2357768432c9a5f3258ebfe1108, State: Running, Role: LEADER
I20260810 15:20:39.515075 6996 consensus_queue.cc:237] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } }
I20260810 15:20:39.515672 6320 catalog_manager.cc:5495] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 reported cstate change: term changed from 0 to 1, leader changed from <none> to c657f2357768432c9a5f3258ebfe1108 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "c657f2357768432c9a5f3258ebfe1108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c657f2357768432c9a5f3258ebfe1108" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 } health_report { overall_health: UNKNOWN } } }
I20260810 15:20:39.522461 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:39.520027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.4.64.65:45955)
0810 15:20:39.520177 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:39.520179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:39.520187 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:39.520206 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:39.520206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:39.520216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:39.520331 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:39.520443 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:39.520936 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:39.521781 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:39.522001 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:39.522057 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:39.522221 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:39.522270 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:39.522321 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:39.522378 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 15:20:39.522398 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260810 15:20:39.525156 6649 raft_consensus.cc:1261] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Refusing update from remote peer c657f2357768432c9a5f3258ebfe1108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:39.525177 6804 raft_consensus.cc:1261] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Refusing update from remote peer c657f2357768432c9a5f3258ebfe1108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:39.525381 7008 consensus_queue.cc:1035] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5711d0ebbd344ff80cfd64e8e6b0078" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:39.525507 6996 consensus_queue.cc:1035] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fbb6c37f8bd4e03ac127185d1ffa5bc" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:20:39.530207 6956 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 15:20:41.048977 6319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:41.049086 6319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:41.053457 6319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 210407ab55da425c955abb8552f5db87: Sending DeleteTablet for 3 replicas of tablet 2012d9ebc5cc45589ad8cc55ba3ce165
I20260810 15:20:41.053833 6781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2012d9ebc5cc45589ad8cc55ba3ce165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:41 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57632
I20260810 15:20:41.053851 6626 tablet_service.cc:1514] Processing DeleteTablet for tablet 2012d9ebc5cc45589ad8cc55ba3ce165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:41 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47548
I20260810 15:20:41.054126 7043 tablet_replica.cc:330] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: stopping tablet replica
I20260810 15:20:41.054128 7041 tablet_replica.cc:330] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: stopping tablet replica
I20260810 15:20:41.054193 7043 raft_consensus.cc:2229] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:41.054236 7043 raft_consensus.cc:2258] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:41.054312 7041 raft_consensus.cc:2229] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:41.054384 7041 raft_consensus.cc:2258] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:41.054606 6469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2012d9ebc5cc45589ad8cc55ba3ce165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:41 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:35512
I20260810 15:20:41.054733 7041 ts_tablet_manager.cc:1905] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:41.055012 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 6399
I20260810 15:20:41.055032 7045 tablet_replica.cc:330] T 2012d9ebc5cc45589ad8cc55ba3ce165 P c657f2357768432c9a5f3258ebfe1108: stopping tablet replica
I20260810 15:20:41.057762 7041 ts_tablet_manager.cc:1918] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:20:41.057852 7041 log.cc:1196] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/2012d9ebc5cc45589ad8cc55ba3ce165
I20260810 15:20:41.058173 7041 ts_tablet_manager.cc:1939] T 2012d9ebc5cc45589ad8cc55ba3ce165 P 9fbb6c37f8bd4e03ac127185d1ffa5bc: Deleting consensus metadata
I20260810 15:20:41.064483 6305 catalog_manager.cc:4841] TS 9fbb6c37f8bd4e03ac127185d1ffa5bc (127.4.64.67:43899): tablet 2012d9ebc5cc45589ad8cc55ba3ce165 (table test-table [id=b917974890be4a0095459f9c6dc5328b]) successfully deleted
I20260810 15:20:41.066073 7043 ts_tablet_manager.cc:1905] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:41.068249 7043 ts_tablet_manager.cc:1918] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:20:41.068325 7043 log.cc:1196] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/2012d9ebc5cc45589ad8cc55ba3ce165
I20260810 15:20:41.068625 7043 ts_tablet_manager.cc:1939] T 2012d9ebc5cc45589ad8cc55ba3ce165 P d5711d0ebbd344ff80cfd64e8e6b0078: Deleting consensus metadata
I20260810 15:20:41.069046 6305 catalog_manager.cc:4841] TS d5711d0ebbd344ff80cfd64e8e6b0078 (127.4.64.66:39655): tablet 2012d9ebc5cc45589ad8cc55ba3ce165 (table test-table [id=b917974890be4a0095459f9c6dc5328b]) successfully deleted
W20260810 15:20:41.073388 6302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:45955 (error 108)
I20260810 15:20:41.073699 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 6553
I20260810 15:20:41.078869 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:41.073845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45955 (local address 127.0.0.1:35540)
0810 15:20:41.078762 (+ 4917us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:41.078823 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4889,"thread_start_us":73,"threads_started":1}
W20260810 15:20:41.079062 6302 catalog_manager.cc:4584] TS c657f2357768432c9a5f3258ebfe1108 (127.4.64.65:45955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2012d9ebc5cc45589ad8cc55ba3ce165: Network error: Client connection negotiation failed: client connection to 127.4.64.65:45955: connect: Connection refused (error 111)
I20260810 15:20:41.083503 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 6710
I20260810 15:20:41.091854 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 6285
2026-08-10T15:20:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5779 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:41 dist-test-slave-9zdj krb5kdc[7050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:41 dist-test-slave-9zdj krb5kdc[7050](info): set up 2 sockets
Aug 10 15:20:41 dist-test-slave-9zdj krb5kdc[7050](info): commencing operation
krb5kdc: starting...
W20260810 15:20:42.231388 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:42.368069 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:36887
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:36887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:42.452942 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:42.453177 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:42.453202 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:42.454439 7066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:42.454483 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:42.454499 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:42.454511 7066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:42.454566 7066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:42.456055 7066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:36887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:36887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:42.456367 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:42.456629 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:42.462147 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:42.462143 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:42.462164 7066 server_base.cc:1034] running on GCE node
W20260810 15:20:42.462175 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:42.462636 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:42.462972 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:42.464159 7066 hybrid_clock.cc:648] HybridClock initialized: now 1786375242464107 us; error 78 us; skew 500 ppm
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:42.471591 7066 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:42.472147 7066 webserver.cc:460] Webserver started at http://127.4.64.126:35193/ using document root <none> and password file <none>
I20260810 15:20:42.472393 7066 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:42.472467 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:42.472623 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:42.473841 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e1f56212a26a4f87ab2f928b319ccb45"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "cad9d0798cde6e579275dd73685699f5"
server_key_iv: "ae05908fcc2b60b50001bd8161b2c601"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.474215 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e1f56212a26a4f87ab2f928b319ccb45"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "cad9d0798cde6e579275dd73685699f5"
server_key_iv: "ae05908fcc2b60b50001bd8161b2c601"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.475584 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:42.476329 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.476504 7066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:42.476613 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e1f56212a26a4f87ab2f928b319ccb45"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "cad9d0798cde6e579275dd73685699f5"
server_key_iv: "ae05908fcc2b60b50001bd8161b2c601"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.476688 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:42.493172 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:42.495363 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:42.495537 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:42.500092 7066 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:36887
I20260810 15:20:42.500134 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:36887 every 8 connection(s)
I20260810 15:20:42.500535 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:42.501456 7141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:42.503801 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45: Bootstrap starting.
I20260810 15:20:42.504104 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7066
I20260810 15:20:42.504206 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:42.504391 4353 external_mini_cluster.cc:1428] Setting key e0f3fa53a6f4447db85ff759427cb3df
I20260810 15:20:42.504421 7141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:42.504763 7141 log.cc:824] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:42.505679 7141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45: No bootstrap required, opened a new log
I20260810 15:20:42.508055 7141 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f56212a26a4f87ab2f928b319ccb45" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36887 } }
I20260810 15:20:42.508237 7141 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:42.508285 7141 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f56212a26a4f87ab2f928b319ccb45, State: Initialized, Role: FOLLOWER
I20260810 15:20:42.508462 7141 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f56212a26a4f87ab2f928b319ccb45" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36887 } }
I20260810 15:20:42.508558 7141 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:42.508605 7141 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:42.508677 7141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:42.509297 7141 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f56212a26a4f87ab2f928b319ccb45" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36887 } }
I20260810 15:20:42.509413 7141 leader_election.cc:302] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1f56212a26a4f87ab2f928b319ccb45; no voters:
I20260810 15:20:42.509578 7141 leader_election.cc:288] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:42.509682 7147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:42.509858 7141 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:42.509873 7147 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [term 1 LEADER]: Becoming Leader. State: Replica: e1f56212a26a4f87ab2f928b319ccb45, State: Running, Role: LEADER
I20260810 15:20:42.509989 7147 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f56212a26a4f87ab2f928b319ccb45" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36887 } }
I20260810 15:20:42.510356 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1f56212a26a4f87ab2f928b319ccb45. Latest consensus state: current_term: 1 leader_uuid: "e1f56212a26a4f87ab2f928b319ccb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f56212a26a4f87ab2f928b319ccb45" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36887 } } }
I20260810 15:20:42.510422 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:42.511243 7149 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1f56212a26a4f87ab2f928b319ccb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f56212a26a4f87ab2f928b319ccb45" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 36887 } } }
I20260810 15:20:42.511565 7149 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:42.511673 7154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:42.512290 7154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:42.514249 7154 catalog_manager.cc:1347] Generated new cluster ID: 9b0a4cfedc8c452290a1eefb2873682c
I20260810 15:20:42.514304 7154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:42.517182 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:42.504928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.4.64.126:36887)
0810 15:20:42.505153 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:42.505164 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:42.505203 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:20:42.505852 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:42.505861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:42.505923 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:42.506081 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:42.506721 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.508196 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.509220 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.509903 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.510580 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:42.513388 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:42.513391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:42.513391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:42.513402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:42.515448 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:42.515890 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:42.515893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:42.515893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:42.516007 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:42.516242 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:42.516243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:42.516243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:42.516550 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:42.516678 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:42.516844 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 15:20:42.516972 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260810 15:20:42.549119 7154 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:42.549806 7154 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:42.563191 7154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1f56212a26a4f87ab2f928b319ccb45: Generated new TSK 0
I20260810 15:20:42.563444 7154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:42.606667 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36887
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:42.694649 7171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:42.694903 7171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:42.694945 7171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:42.696216 7171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:42.696288 7171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:42.696331 7171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:42.697870 7171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:42.698159 7171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:42.698438 7171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:42.704140 7176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:42.704133 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:42.704133 7178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:42.704341 7171 server_base.cc:1034] running on GCE node
I20260810 15:20:42.704659 7171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:42.704883 7171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:42.706022 7171 hybrid_clock.cc:648] HybridClock initialized: now 1786375242706003 us; error 34 us; skew 500 ppm
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:42.708688 7171 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:42.709141 7171 webserver.cc:460] Webserver started at http://127.4.64.65:34305/ using document root <none> and password file <none>
I20260810 15:20:42.709419 7171 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:42.709470 7171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:42.709688 7171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:42.710882 7171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "f06c0f084f3d4c488be41ef2b7b4baaf"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "4af1ea77c8178385e8a0fde545387b9b"
server_key_iv: "9e0749634deb72ec9fb8a6581cd2b6b9"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.711278 7171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "f06c0f084f3d4c488be41ef2b7b4baaf"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "4af1ea77c8178385e8a0fde545387b9b"
server_key_iv: "9e0749634deb72ec9fb8a6581cd2b6b9"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.712815 7171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:42.713698 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.713940 7171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:42.714061 7171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "f06c0f084f3d4c488be41ef2b7b4baaf"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "4af1ea77c8178385e8a0fde545387b9b"
server_key_iv: "9e0749634deb72ec9fb8a6581cd2b6b9"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.714169 7171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:42.744427 7171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:42.747417 7171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:42.747625 7171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:42.747913 7171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:42.748281 7171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:42.748342 7171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.748391 7171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:42.748418 7171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.755429 7171 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:46691
I20260810 15:20:42.755478 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:46691 every 8 connection(s)
I20260810 15:20:42.755864 7171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:42.764465 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7171
I20260810 15:20:42.764575 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:42.764781 4353 external_mini_cluster.cc:1428] Setting key 60dbc05de23da9afc28ad7cf6f1251b1
I20260810 15:20:42.766319 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:42.756697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:53303 (local address 127.4.64.126:36887)
0810 15:20:42.756779 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:42.756782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:42.757057 (+ 275us) server_negotiation.cc:407] Connection header received
0810 15:20:42.757538 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:42.757539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:42.757550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:42.757605 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:42.758719 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.759233 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.760287 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.760515 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.760605 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:42.763085 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:42.763087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:42.763087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:42.763094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:42.764459 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:42.765083 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:42.765085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:42.765085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:42.765181 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:42.765492 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:42.765493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:42.765493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:42.765681 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:42.765778 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:42.766121 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 15:20:42.766228 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 15:20:42.766525 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:42.756663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36887 (local address 127.4.64.65:53303)
0810 15:20:42.756906 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:42.756949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:42.757446 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:42.757742 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:42.757748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:42.758408 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:42.758613 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:42.758618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.759326 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.759327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:42.760198 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:42.760201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.760646 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.760647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:42.760784 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:42.761387 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:42.761396 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:42.762985 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:42.764681 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:42.764689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:42.764692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:42.764996 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:42.765253 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:42.765254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:42.765254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:42.765400 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:42.765774 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:42.765782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:42.765927 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 15:20:42.766056 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 15:20:42.766249 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":124,"threads_started":1}
I20260810 15:20:42.767114 7316 heartbeater.cc:347] Connected to a master server at 127.4.64.126:36887
I20260810 15:20:42.767230 7316 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:42.767446 7316 heartbeater.cc:511] Master 127.4.64.126:36887 requested a full tablet report, sending...
I20260810 15:20:42.768093 7100 ts_manager.cc:194] Registered new tserver with Master: f06c0f084f3d4c488be41ef2b7b4baaf (127.4.64.65:46691)
I20260810 15:20:42.768715 7100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:53303
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:42.794927 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36887
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:42.882436 7321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:42.882653 7321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:42.882680 7321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:42.883880 7321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:42.883925 7321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:42.883978 7321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:42.885550 7321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:42.885852 7321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:42.886106 7321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:42.891582 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:42.891609 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:42.891587 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:42.891810 7321 server_base.cc:1034] running on GCE node
I20260810 15:20:42.892079 7321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:42.892354 7321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:42.893564 7321 hybrid_clock.cc:648] HybridClock initialized: now 1786375242893547 us; error 60 us; skew 500 ppm
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:42.896595 7321 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:42.897145 7321 webserver.cc:460] Webserver started at http://127.4.64.66:40391/ using document root <none> and password file <none>
I20260810 15:20:42.897428 7321 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:42.897506 7321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:42.897707 7321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:42.898931 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "a352821362fd4d9282de7c3fbce49339"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "1433a7e40484f7d5f41e18afa6034801"
server_key_iv: "46a8ecff652a0f834e56624bba447d12"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.899351 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "a352821362fd4d9282de7c3fbce49339"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "1433a7e40484f7d5f41e18afa6034801"
server_key_iv: "46a8ecff652a0f834e56624bba447d12"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.901120 7321 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 15:20:42.902114 7338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.902371 7321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:42.902498 7321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "a352821362fd4d9282de7c3fbce49339"
format_stamp: "Formatted at 2026-08-10 15:20:42 on dist-test-slave-9zdj"
server_key: "1433a7e40484f7d5f41e18afa6034801"
server_key_iv: "46a8ecff652a0f834e56624bba447d12"
server_key_version: "encryptionkey@0"
I20260810 15:20:42.902582 7321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:42.915537 7321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:42.918287 7321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:42.918445 7321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:42.918704 7321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:42.919104 7321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:42.919142 7321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.919163 7321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:42.919196 7321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:42.925218 7321 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:34587
I20260810 15:20:42.925304 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:34587 every 8 connection(s)
I20260810 15:20:42.925699 7321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:42.931635 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7321
I20260810 15:20:42.931758 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:42.932013 4353 external_mini_cluster.cc:1428] Setting key 3e198dce2eaeddffde3432858c29622b
Aug 10 15:20:42 dist-test-slave-9zdj krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375242, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:42.935856 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:42.926473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:34561 (local address 127.4.64.126:36887)
0810 15:20:42.926537 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:42.926539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:42.926744 (+ 205us) server_negotiation.cc:407] Connection header received
0810 15:20:42.927360 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:42.927362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:42.927375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:42.927419 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:42.928436 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.928959 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.930079 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.930297 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.930422 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:42.932869 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:42.932871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:42.932871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:42.932879 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:42.934277 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:42.934819 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:42.934820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:42.934820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:42.934886 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:42.935185 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:42.935186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:42.935186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:42.935298 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:42.935374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:42.935708 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 15:20:42.935784 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:20:42.936249 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:42.926453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36887 (local address 127.4.64.66:34561)
0810 15:20:42.926608 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:42.926654 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:42.927192 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:42.927500 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:42.927504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:42.928117 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:42.928326 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:42.928331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.929068 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.929069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:42.929989 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:42.929991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:42.930408 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:42.930409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:42.930556 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:42.931031 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:42.931040 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:42.932774 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:42.934362 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:42.934369 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:42.934372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:42.934734 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:42.934960 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:42.934961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:42.934962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:42.935116 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:42.935367 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:42.935374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:42.935562 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 15:20:42.935687 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 15:20:42.935894 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260810 15:20:42.936866 7461 heartbeater.cc:347] Connected to a master server at 127.4.64.126:36887
I20260810 15:20:42.936962 7461 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:42.937166 7461 heartbeater.cc:511] Master 127.4.64.126:36887 requested a full tablet report, sending...
I20260810 15:20:42.937602 7100 ts_manager.cc:194] Registered new tserver with Master: a352821362fd4d9282de7c3fbce49339 (127.4.64.66:34587)
I20260810 15:20:42.938040 7100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:34561
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:42.961680 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36887
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:43.049412 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:43.049650 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:43.049678 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:43.050940 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:43.050988 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:43.051044 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:43.053735 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:36887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:43.054044 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:43.054318 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:43.060959 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:43.060950 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:43.060949 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:43.061131 7467 server_base.cc:1034] running on GCE node
I20260810 15:20:43.061365 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:43.061682 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:43.062878 7467 hybrid_clock.cc:648] HybridClock initialized: now 1786375243062846 us; error 49 us; skew 500 ppm
Aug 10 15:20:43 dist-test-slave-9zdj krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375243, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:43.065835 7467 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:43.066318 7467 webserver.cc:460] Webserver started at http://127.4.64.67:45035/ using document root <none> and password file <none>
I20260810 15:20:43.066586 7467 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:43.066660 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:43.066831 7467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:43.067991 7467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "09266aaf25434677b02504785c5071e4"
format_stamp: "Formatted at 2026-08-10 15:20:43 on dist-test-slave-9zdj"
server_key: "d3a961bd74e61865ea2379f21785e9a0"
server_key_iv: "6044454d7d0b45643e7606538fdabc6b"
server_key_version: "encryptionkey@0"
I20260810 15:20:43.068377 7467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "09266aaf25434677b02504785c5071e4"
format_stamp: "Formatted at 2026-08-10 15:20:43 on dist-test-slave-9zdj"
server_key: "d3a961bd74e61865ea2379f21785e9a0"
server_key_iv: "6044454d7d0b45643e7606538fdabc6b"
server_key_version: "encryptionkey@0"
I20260810 15:20:43.069839 7467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:20:43.070657 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:43.070845 7467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:43.070955 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "09266aaf25434677b02504785c5071e4"
format_stamp: "Formatted at 2026-08-10 15:20:43 on dist-test-slave-9zdj"
server_key: "d3a961bd74e61865ea2379f21785e9a0"
server_key_iv: "6044454d7d0b45643e7606538fdabc6b"
server_key_version: "encryptionkey@0"
I20260810 15:20:43.071033 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:43.088637 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:43.090961 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:43.091151 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:43.091471 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:43.091826 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:43.091886 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:43.091923 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:43.091969 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:43.098495 7467 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:36531
I20260810 15:20:43.098548 7612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:36531 every 8 connection(s)
I20260810 15:20:43.098903 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:20:43 dist-test-slave-9zdj krb5kdc[7050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375243, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:43.108314 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7467
I20260810 15:20:43.108455 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:43.108655 4353 external_mini_cluster.cc:1428] Setting key f9834b975ecc324fc00953d83dafc38a
I20260810 15:20:43.110070 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:43.099735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:33967 (local address 127.4.64.126:36887)
0810 15:20:43.099836 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:43.099838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:43.100034 (+ 196us) server_negotiation.cc:407] Connection header received
0810 15:20:43.100714 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:43.100715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:43.100729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:43.100809 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:43.102004 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:43.102537 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:43.103619 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:43.103827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:43.103921 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:43.106470 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:43.106472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:43.106472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:43.106481 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:43.108280 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:43.108905 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:43.108907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:43.108907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:43.108979 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:43.109267 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:43.109268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:43.109268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:43.109401 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:43.109477 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:43.109856 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 15:20:43.109963 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:20:43.110419 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:43.099690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:36887 (local address 127.4.64.67:33967)
0810 15:20:43.099892 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:43.099944 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:43.100584 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:43.100890 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:43.100894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:43.101578 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:43.101893 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:43.101899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:43.102655 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:43.102657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:43.103511 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:43.103513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:43.103935 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:43.103936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:43.104072 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:43.104633 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:43.104648 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:43.106347 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:43.108397 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:43.108411 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:43.108413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:43.108791 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:43.109053 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:43.109054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:43.109055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:43.109184 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:43.109480 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:43.109488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:43.109697 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 15:20:43.109832 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 15:20:43.110066 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
I20260810 15:20:43.110975 7613 heartbeater.cc:347] Connected to a master server at 127.4.64.126:36887
I20260810 15:20:43.111181 7613 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:43.111398 7613 heartbeater.cc:511] Master 127.4.64.126:36887 requested a full tablet report, sending...
I20260810 15:20:43.111860 7100 ts_manager.cc:194] Registered new tserver with Master: 09266aaf25434677b02504785c5071e4 (127.4.64.67:36531)
I20260810 15:20:43.112265 7100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:33967
I20260810 15:20:43.120925 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 15:20:43.135416 7146 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:43.134323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.4.64.126:36887)
0810 15:20:43.134389 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:43.134391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:43.134541 (+ 150us) server_negotiation.cc:407] Connection header received
0810 15:20:43.134608 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:43.134609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:43.134619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:43.134685 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:43.135304 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46288: BlockingRecv error: recv got EOF from 127.0.0.1:46288 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:20:43.136549 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7171
I20260810 15:20:43.146754 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7321
I20260810 15:20:43.154981 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7467
I20260810 15:20:43.165580 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7066
2026-08-10T15:20:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2073 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:43 dist-test-slave-9zdj krb5kdc[7625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:43 dist-test-slave-9zdj krb5kdc[7625](info): set up 2 sockets
Aug 10 15:20:43 dist-test-slave-9zdj krb5kdc[7625](info): commencing operation
krb5kdc: starting...
W20260810 15:20:44.310786 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:44 dist-test-slave-9zdj krb5kdc[7625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:44.453976 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37169
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:37169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:44.540630 7639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:44.540913 7639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:44.540953 7639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:44.542189 7639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:44.542272 7639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:44.542296 7639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:44.542316 7639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:44.542348 7639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:44.543915 7639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:37169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:44.544229 7639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:44.544474 7639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:44.549823 7646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:44.549865 7647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:44.549865 7649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:44.550083 7639 server_base.cc:1034] running on GCE node
I20260810 15:20:44.550550 7639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:44.550957 7639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:44.552227 7639 hybrid_clock.cc:648] HybridClock initialized: now 1786375244552210 us; error 44 us; skew 500 ppm
Aug 10 15:20:44 dist-test-slave-9zdj krb5kdc[7625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375244, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:44.555207 7639 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:44.555639 7639 webserver.cc:460] Webserver started at http://127.4.64.126:41561/ using document root <none> and password file <none>
I20260810 15:20:44.555833 7639 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:44.555874 7639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:44.555971 7639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:44.557062 7639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "1cd690b5a0984c01babee754ac94f7b7"
format_stamp: "Formatted at 2026-08-10 15:20:44 on dist-test-slave-9zdj"
server_key: "0069a36dbc3d3be3bb3dead6b7ca1f29"
server_key_iv: "701690947445284496a6a7ecb6b42be6"
server_key_version: "encryptionkey@0"
I20260810 15:20:44.557390 7639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "1cd690b5a0984c01babee754ac94f7b7"
format_stamp: "Formatted at 2026-08-10 15:20:44 on dist-test-slave-9zdj"
server_key: "0069a36dbc3d3be3bb3dead6b7ca1f29"
server_key_iv: "701690947445284496a6a7ecb6b42be6"
server_key_version: "encryptionkey@0"
I20260810 15:20:44.558840 7639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:20:44.559610 7656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:44.559772 7639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:44.559857 7639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "1cd690b5a0984c01babee754ac94f7b7"
format_stamp: "Formatted at 2026-08-10 15:20:44 on dist-test-slave-9zdj"
server_key: "0069a36dbc3d3be3bb3dead6b7ca1f29"
server_key_iv: "701690947445284496a6a7ecb6b42be6"
server_key_version: "encryptionkey@0"
I20260810 15:20:44.559935 7639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:44.594949 7639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:44.597079 7639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:44.597266 7639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:44.601876 7639 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:37169
I20260810 15:20:44.601920 7711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:37169 every 8 connection(s)
I20260810 15:20:44.602298 7639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:44.603199 7712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:44.605813 7712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7: Bootstrap starting.
I20260810 15:20:44.606485 7712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:44.606815 7712 log.cc:824] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:44.607576 7712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7: No bootstrap required, opened a new log
I20260810 15:20:44.609304 7712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd690b5a0984c01babee754ac94f7b7" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37169 } }
I20260810 15:20:44.609459 7712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:44.609483 7712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd690b5a0984c01babee754ac94f7b7, State: Initialized, Role: FOLLOWER
I20260810 15:20:44.609701 7712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd690b5a0984c01babee754ac94f7b7" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37169 } }
I20260810 15:20:44.609778 7712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:44.609802 7712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:44.609874 7712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:44.610765 7712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd690b5a0984c01babee754ac94f7b7" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37169 } }
I20260810 15:20:44.610800 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7639
I20260810 15:20:44.610932 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:44.610903 7712 leader_election.cc:302] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cd690b5a0984c01babee754ac94f7b7; no voters:
I20260810 15:20:44.611120 4353 external_mini_cluster.cc:1428] Setting key 2a438947961711c99117c0fc9de03503
I20260810 15:20:44.611133 7712 leader_election.cc:288] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:44.611167 7715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:44.611358 7715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd690b5a0984c01babee754ac94f7b7, State: Running, Role: LEADER
I20260810 15:20:44.611438 7712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:44.611521 7715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd690b5a0984c01babee754ac94f7b7" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37169 } }
I20260810 15:20:44.611990 7716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cd690b5a0984c01babee754ac94f7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd690b5a0984c01babee754ac94f7b7" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37169 } } }
I20260810 15:20:44.612088 7716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:44.612078 7718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cd690b5a0984c01babee754ac94f7b7. Latest consensus state: current_term: 1 leader_uuid: "1cd690b5a0984c01babee754ac94f7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd690b5a0984c01babee754ac94f7b7" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37169 } } }
I20260810 15:20:44.612370 7730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:44.612449 7718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:44.613384 7730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:44.615087 7730 catalog_manager.cc:1347] Generated new cluster ID: bf349e7751814cf28ae8d23694d64e13
I20260810 15:20:44.615144 7730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:20:44 dist-test-slave-9zdj krb5kdc[7625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:44.624898 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:44.611728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.4.64.126:37169)
0810 15:20:44.611952 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:44.611962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:44.612001 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:20:44.612229 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:44.612238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:44.612306 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:44.612467 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:44.613534 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:44.615103 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:44.616247 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:44.617018 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:44.617145 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:44.620246 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:44.620251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:44.620251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:44.620266 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:44.622981 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:44.623432 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:44.623435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:44.623436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:44.623579 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:44.623811 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:44.623812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:44.623812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:44.624208 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:44.624423 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:44.624497 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 15:20:44.624658 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260810 15:20:44.626312 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:44.629467 7730 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:44.630008 7730 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:44.636312 7730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cd690b5a0984c01babee754ac94f7b7: Generated new TSK 0
I20260810 15:20:44.636543 7730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:44.645232 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:44.640997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.4.64.126:37169)
0810 15:20:44.641052 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:44.641054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:44.641154 (+ 100us) server_negotiation.cc:407] Connection header received
0810 15:20:44.641342 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:44.641343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:44.641356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:44.641412 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:44.641940 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:44.642794 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:44.643593 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:44.643845 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:44.643928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:44.644713 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:44.644714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:44.644714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:44.644932 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:44.644950 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:44.644957 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:44.645029 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:44.645097 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 15:20:44.645137 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:20:44.648350 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7639
2026-08-10T15:20:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1481 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:44 dist-test-slave-9zdj krb5kdc[7750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:44 dist-test-slave-9zdj krb5kdc[7750](info): set up 2 sockets
Aug 10 15:20:44 dist-test-slave-9zdj krb5kdc[7750](info): commencing operation
krb5kdc: starting...
W20260810 15:20:45.772670 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:45 dist-test-slave-9zdj krb5kdc[7750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:45.917549 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34641
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:46.004467 7765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:46.004739 7765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:46.004781 7765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:46.006068 7765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:46.006145 7765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:46.006167 7765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:46.006188 7765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:46.006222 7765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:46.007795 7765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:46.008091 7765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:46.008361 7765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:46.013653 7773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.013830 7770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.013883 7771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:46.014109 7765 server_base.cc:1034] running on GCE node
I20260810 15:20:46.014375 7765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:46.014711 7765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:46.015965 7765 hybrid_clock.cc:648] HybridClock initialized: now 1786375246015945 us; error 47 us; skew 500 ppm
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:46.019006 7765 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:46.019541 7765 webserver.cc:460] Webserver started at http://127.4.64.126:40559/ using document root <none> and password file <none>
I20260810 15:20:46.019775 7765 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:46.019819 7765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:46.019917 7765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:46.021009 7765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "719aec38d19b1355ee8caab11067c9f0"
server_key_iv: "a56ba5ccfe139b1d9181a897b85c2f57"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.021343 7765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "719aec38d19b1355ee8caab11067c9f0"
server_key_iv: "a56ba5ccfe139b1d9181a897b85c2f57"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.022855 7765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:46.023694 7780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.023949 7765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:46.024050 7765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "719aec38d19b1355ee8caab11067c9f0"
server_key_iv: "a56ba5ccfe139b1d9181a897b85c2f57"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.024130 7765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:46.035228 7765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:46.037547 7765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:46.037758 7765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:46.041790 7765 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34641
I20260810 15:20:46.041882 7840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34641 every 8 connection(s)
I20260810 15:20:46.042241 7765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:46.043383 7841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:46.044313 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7765
I20260810 15:20:46.044404 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:46.044626 4353 external_mini_cluster.cc:1428] Setting key 5bb0c612fbb1397fc4a6809b3a4de3da
I20260810 15:20:46.046356 7841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6: Bootstrap starting.
I20260810 15:20:46.047183 7841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:46.047560 7841 log.cc:824] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:46.048324 7841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6: No bootstrap required, opened a new log
I20260810 15:20:46.050983 7841 raft_consensus.cc:348] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34641 } }
I20260810 15:20:46.051183 7841 raft_consensus.cc:374] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:46.051229 7841 raft_consensus.cc:729] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46ae8143cbc45eca00a1cb1bccc3bc6, State: Initialized, Role: FOLLOWER
I20260810 15:20:46.051401 7841 consensus_queue.cc:260] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34641 } }
I20260810 15:20:46.051513 7841 raft_consensus.cc:388] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:46.051548 7841 raft_consensus.cc:482] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:46.051596 7841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:46.052299 7841 raft_consensus.cc:504] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34641 } }
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:46.052397 7841 leader_election.cc:302] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d46ae8143cbc45eca00a1cb1bccc3bc6; no voters:
I20260810 15:20:46.052594 7841 leader_election.cc:288] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:46.052672 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:46.052892 7847 raft_consensus.cc:686] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [term 1 LEADER]: Becoming Leader. State: Replica: d46ae8143cbc45eca00a1cb1bccc3bc6, State: Running, Role: LEADER
I20260810 15:20:46.052896 7841 sys_catalog.cc:564] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:46.053066 7847 consensus_queue.cc:237] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34641 } }
I20260810 15:20:46.053719 7850 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d46ae8143cbc45eca00a1cb1bccc3bc6. Latest consensus state: current_term: 1 leader_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34641 } } }
I20260810 15:20:46.053817 7850 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:46.053948 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d46ae8143cbc45eca00a1cb1bccc3bc6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34641 } } }
I20260810 15:20:46.054044 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:46.054754 7854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:46.055292 7854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:46.057227 7854 catalog_manager.cc:1347] Generated new cluster ID: 42004c1307014ffd9bd22ed8a68f22b7
I20260810 15:20:46.057282 7854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:46.057682 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.045283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.4.64.126:34641)
0810 15:20:46.045504 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:46.045513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:46.045542 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:20:46.046158 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:46.046168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:46.046238 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:46.046379 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:46.047339 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.048339 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.049467 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.050058 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.050145 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.053016 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:46.053021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:46.053021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:46.053036 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:46.055652 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.056121 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.056124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.056125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.056240 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.056484 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.056485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.056485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.056768 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:46.056883 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:46.057292 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 15:20:46.057438 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":121,"threads_started":1}
I20260810 15:20:46.074775 7854 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:46.075584 7854 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:46.085155 7854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d46ae8143cbc45eca00a1cb1bccc3bc6: Generated new TSK 0
I20260810 15:20:46.085421 7854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:46.117908 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34641
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:46.205013 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:46.205271 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:46.205297 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:46.206602 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:46.206656 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:46.206686 7875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:46.208271 7875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:46.208554 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:46.208822 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:46.214378 7884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.214422 7882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.214404 7881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:46.214931 7875 server_base.cc:1034] running on GCE node
I20260810 15:20:46.215132 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:46.215404 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:46.216590 7875 hybrid_clock.cc:648] HybridClock initialized: now 1786375246216576 us; error 32 us; skew 500 ppm
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:46.219578 7875 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:46.220080 7875 webserver.cc:460] Webserver started at http://127.4.64.65:37213/ using document root <none> and password file <none>
I20260810 15:20:46.220351 7875 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:46.220404 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:46.220543 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:46.221740 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "676b30f902d64cdba54cfa7238ff7ecc"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "1a2c9f2643adbb4e91893c066207b224"
server_key_iv: "d37d73527f7663430033b3252c913f66"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.222111 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "676b30f902d64cdba54cfa7238ff7ecc"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "1a2c9f2643adbb4e91893c066207b224"
server_key_iv: "d37d73527f7663430033b3252c913f66"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.223555 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:46.224457 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.224768 7875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:46.224910 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "676b30f902d64cdba54cfa7238ff7ecc"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "1a2c9f2643adbb4e91893c066207b224"
server_key_iv: "d37d73527f7663430033b3252c913f66"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.224989 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:46.247792 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:46.250643 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:46.250897 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:46.251199 7875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:46.251616 7875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:46.251686 7875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.251739 7875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:46.251775 7875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.259394 7875 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:34809
I20260810 15:20:46.259464 8023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:34809 every 8 connection(s)
I20260810 15:20:46.259871 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:46.264623 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 7875
I20260810 15:20:46.264781 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:46.264988 4353 external_mini_cluster.cc:1428] Setting key 3006b50c69879164bba3162c482d980e
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:46.270692 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.260700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:59019 (local address 127.4.64.126:34641)
0810 15:20:46.260751 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:46.260753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:46.261000 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:20:46.261502 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:46.261503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:46.261515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:46.261585 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:46.262628 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.263203 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.264287 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.264505 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.264609 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.267545 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:46.267547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:46.267547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:46.267555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:46.269105 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.269554 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.269555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.269555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.269667 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.269952 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.269953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.269953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.270083 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:46.270155 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:46.270485 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 15:20:46.270585 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:46.270906 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.260664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34641 (local address 127.4.64.65:59019)
0810 15:20:46.260855 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:46.260902 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:46.261406 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:46.261691 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:46.261695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:46.262318 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:46.262527 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:46.262532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.263303 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.263304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:46.264182 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:46.264185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.264622 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.264624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:46.264775 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.265503 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:46.265512 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:46.267412 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:46.269217 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:46.269226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:46.269228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:46.269471 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:46.269740 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:46.269740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:46.269741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:46.269865 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:46.270165 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:46.270173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:46.270333 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 15:20:46.270451 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 15:20:46.270624 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260810 15:20:46.271521 8024 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34641
I20260810 15:20:46.271677 8024 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:46.271920 8024 heartbeater.cc:511] Master 127.4.64.126:34641 requested a full tablet report, sending...
I20260810 15:20:46.272603 7800 ts_manager.cc:194] Registered new tserver with Master: 676b30f902d64cdba54cfa7238ff7ecc (127.4.64.65:34809)
I20260810 15:20:46.273736 7800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:59019
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:46.297959 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34641
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:46.384228 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:46.384450 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:46.384476 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:46.385733 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:46.385782 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:46.385810 8030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:46.387336 8030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:46.387615 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:46.387863 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:46.393383 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.393330 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:46.393488 8030 server_base.cc:1034] running on GCE node
W20260810 15:20:46.393647 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:46.393868 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:46.394131 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:46.395300 8030 hybrid_clock.cc:648] HybridClock initialized: now 1786375246395283 us; error 44 us; skew 500 ppm
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:46.398357 8030 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:46.398881 8030 webserver.cc:460] Webserver started at http://127.4.64.66:41241/ using document root <none> and password file <none>
I20260810 15:20:46.399134 8030 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:46.399205 8030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:46.399367 8030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:46.400488 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "0f9e76220bfc4ac984d0f53d35beb004"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "b8d0892faad9ac0b73c5c510f67a46ce"
server_key_iv: "4410b49d5ea55fd92f93121725b2621c"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.400872 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "0f9e76220bfc4ac984d0f53d35beb004"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "b8d0892faad9ac0b73c5c510f67a46ce"
server_key_iv: "4410b49d5ea55fd92f93121725b2621c"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.402472 8030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:46.403281 8046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.403503 8030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:46.403644 8030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "0f9e76220bfc4ac984d0f53d35beb004"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "b8d0892faad9ac0b73c5c510f67a46ce"
server_key_iv: "4410b49d5ea55fd92f93121725b2621c"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.403721 8030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:46.420089 8030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:46.422479 8030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:46.422634 8030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:46.422912 8030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:46.423272 8030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:46.423344 8030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.423384 8030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:46.423413 8030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.430120 8030 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:39897
I20260810 15:20:46.430174 8175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:39897 every 8 connection(s)
I20260810 15:20:46.430557 8030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:46.434473 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8030
I20260810 15:20:46.434604 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:46.434830 4353 external_mini_cluster.cc:1428] Setting key 92faa30580f3862159efef3adc506ce4
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:46.442533 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.431557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:55893 (local address 127.4.64.126:34641)
0810 15:20:46.431647 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:46.431649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:46.431914 (+ 265us) server_negotiation.cc:407] Connection header received
0810 15:20:46.432410 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:46.432411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:46.432421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:46.432483 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:46.433706 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.434319 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.435900 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.436096 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.436154 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.439164 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:46.439166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:46.439167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:46.439174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:46.440701 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.441326 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.441327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.441327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.441399 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.441775 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.441776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.441776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.441908 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:46.441967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:46.442380 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 15:20:46.442461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 15:20:46.442896 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.431520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34641 (local address 127.4.64.66:55893)
0810 15:20:46.431732 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:46.431817 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:46.432321 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:46.432561 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:46.432565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:46.433215 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:46.433470 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:46.433476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.434457 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.434458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:46.435803 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:46.435807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.436191 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.436192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:46.436348 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.436949 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:46.436962 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:46.439045 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:46.440817 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:46.440828 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:46.440831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:46.441223 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:46.441477 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:46.441478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:46.441478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:46.441688 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:46.441987 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:46.441995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:46.442206 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 15:20:46.442363 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 15:20:46.442585 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":125,"threads_started":1}
I20260810 15:20:46.443614 8176 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34641
I20260810 15:20:46.443737 8176 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:46.443964 8176 heartbeater.cc:511] Master 127.4.64.126:34641 requested a full tablet report, sending...
I20260810 15:20:46.444437 7800 ts_manager.cc:194] Registered new tserver with Master: 0f9e76220bfc4ac984d0f53d35beb004 (127.4.64.66:39897)
I20260810 15:20:46.444856 7800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:55893
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:46.470726 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34641
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:46.559211 8181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:46.559459 8181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:46.559489 8181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:46.560787 8181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:46.560837 8181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:46.560866 8181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:46.562489 8181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:46.562723 8181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:46.562942 8181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:46.568837 8181 server_base.cc:1034] running on GCE node
W20260810 15:20:46.568784 8190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.568799 8187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:46.568796 8188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:46.569259 8181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:46.569574 8181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:46.570822 8181 hybrid_clock.cc:648] HybridClock initialized: now 1786375246570751 us; error 79 us; skew 500 ppm
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:46.573987 8181 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:46.574529 8181 webserver.cc:460] Webserver started at http://127.4.64.67:42777/ using document root <none> and password file <none>
I20260810 15:20:46.574783 8181 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:46.574855 8181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:46.575022 8181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:46.576170 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "959be52f213943ee86ee80b0181d7b4c"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "acb0b977c5151f823276dc9f97ef217c"
server_key_iv: "cff0f527acbe9ecaed1508dba54dc38d"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.576607 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "959be52f213943ee86ee80b0181d7b4c"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "acb0b977c5151f823276dc9f97ef217c"
server_key_iv: "cff0f527acbe9ecaed1508dba54dc38d"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.578173 8181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:46.579064 8197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.579281 8181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:46.579416 8181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "959be52f213943ee86ee80b0181d7b4c"
format_stamp: "Formatted at 2026-08-10 15:20:46 on dist-test-slave-9zdj"
server_key: "acb0b977c5151f823276dc9f97ef217c"
server_key_iv: "cff0f527acbe9ecaed1508dba54dc38d"
server_key_version: "encryptionkey@0"
I20260810 15:20:46.579500 8181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:46.602226 8181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:46.604898 8181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:46.605103 8181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:46.605427 8181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:46.605886 8181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:46.605950 8181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.605999 8181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:46.606029 8181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:46.613001 8181 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:33395
I20260810 15:20:46.613094 8324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:33395 every 8 connection(s)
I20260810 15:20:46.613510 8181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:46.617681 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8181
I20260810 15:20:46.617800 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:46.617992 4353 external_mini_cluster.cc:1428] Setting key 869a935def3f35a8185cf6b5bdc50b56
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:46.624883 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.614406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:43303 (local address 127.4.64.126:34641)
0810 15:20:46.614468 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:46.614472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:46.614719 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:20:46.615335 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:46.615336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:46.615351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:46.615429 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:46.616439 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.617049 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.618760 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.619074 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.619135 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.621976 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:46.621978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:46.621978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:46.621986 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:46.623500 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.623963 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.623964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.623964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.624022 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.624270 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.624270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.624270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.624414 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:46.624469 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:46.624740 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 15:20:46.624810 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:46.625259 8327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.614384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34641 (local address 127.4.64.67:43303)
0810 15:20:46.614565 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:46.614606 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:46.615225 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:46.615490 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:46.615494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:46.616128 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:46.616344 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:46.616349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.617255 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.617256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:46.618634 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:46.618638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.619158 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.619164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:46.619284 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.619867 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:46.619881 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:46.621862 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:46.623592 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:46.623600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:46.623602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:46.623886 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:46.624091 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:46.624091 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:46.624092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:46.624211 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:46.624475 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:46.624480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:46.624618 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 15:20:46.624712 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 15:20:46.624865 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260810 15:20:46.625980 8325 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34641
I20260810 15:20:46.626103 8325 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:46.626328 8325 heartbeater.cc:511] Master 127.4.64.126:34641 requested a full tablet report, sending...
I20260810 15:20:46.626777 7800 ts_manager.cc:194] Registered new tserver with Master: 959be52f213943ee86ee80b0181d7b4c (127.4.64.67:33395)
I20260810 15:20:46.627373 7800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:43303
I20260810 15:20:46.629988 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[7750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375246, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:46.657912 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:46.649333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.4.64.126:34641)
0810 15:20:46.649460 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:46.649463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:46.649577 (+ 114us) server_negotiation.cc:407] Connection header received
0810 15:20:46.649719 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:46.649720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:46.649731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:46.649813 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:46.650507 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.651052 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.651886 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:46.652090 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:46.652175 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:46.654957 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:46.654958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:46.654959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:46.654969 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:46.656468 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.656948 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.656949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.656949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.657022 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:46.657232 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:46.657233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:46.657233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:46.657412 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:46.657527 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:46.657689 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:20:46.657800 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 15:20:46.658221 7800 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44394
I20260810 15:20:46.658758 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7875
I20260810 15:20:46.672613 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8030
I20260810 15:20:46.683527 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8181
I20260810 15:20:46.692178 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 7765
2026-08-10T15:20:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2043 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[8339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[8339](info): set up 2 sockets
Aug 10 15:20:46 dist-test-slave-9zdj krb5kdc[8339](info): commencing operation
krb5kdc: starting...
W20260810 15:20:47.897197 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:47 dist-test-slave-9zdj krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:48.042793 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:39873
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:48.135790 8353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:48.136071 8353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:48.136111 8353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:48.137367 8353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:48.137452 8353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:48.137476 8353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:48.137496 8353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:48.137526 8353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:48.139212 8353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:48.139528 8353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:48.139799 8353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:48.145366 8359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.145401 8363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.145362 8361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:48.145653 8353 server_base.cc:1034] running on GCE node
I20260810 15:20:48.145998 8353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:48.146389 8353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:48.147593 8353 hybrid_clock.cc:648] HybridClock initialized: now 1786375248147588 us; error 61 us; skew 500 ppm
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:48.163511 8353 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:48.164088 8353 webserver.cc:460] Webserver started at http://127.4.64.126:35489/ using document root <none> and password file <none>
I20260810 15:20:48.164395 8353 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:48.164476 8353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:48.164644 8353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:48.165920 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "12b0129b51e14d8e92864742e52ce477"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "1e81b269544c5702ab2ea5d9a78cd89c"
server_key_iv: "3b99f1af4e848510cc05550b533e0909"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.166319 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "12b0129b51e14d8e92864742e52ce477"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "1e81b269544c5702ab2ea5d9a78cd89c"
server_key_iv: "3b99f1af4e848510cc05550b533e0909"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.167907 8353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:48.168946 8370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.169209 8353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:48.169401 8353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "12b0129b51e14d8e92864742e52ce477"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "1e81b269544c5702ab2ea5d9a78cd89c"
server_key_iv: "3b99f1af4e848510cc05550b533e0909"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.169512 8353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:48.182554 8353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:48.185396 8353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:48.185608 8353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:48.190631 8353 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:39873
I20260810 15:20:48.190734 8429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:39873 every 8 connection(s)
I20260810 15:20:48.191442 8353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:48.192306 8430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:48.195158 8430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477: Bootstrap starting.
I20260810 15:20:48.195894 8430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:48.196269 8430 log.cc:824] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:48.197302 8430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477: No bootstrap required, opened a new log
I20260810 15:20:48.199184 8430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0129b51e14d8e92864742e52ce477" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39873 } }
I20260810 15:20:48.199338 8430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:48.199361 8430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b0129b51e14d8e92864742e52ce477, State: Initialized, Role: FOLLOWER
I20260810 15:20:48.199534 8430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0129b51e14d8e92864742e52ce477" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39873 } }
I20260810 15:20:48.199620 8430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:48.199690 8430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:48.199746 8430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:48.200101 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8353
I20260810 15:20:48.200208 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:48.200582 4353 external_mini_cluster.cc:1428] Setting key 34ab98437e667d2881048ff38da6f2b6
I20260810 15:20:48.200906 8430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0129b51e14d8e92864742e52ce477" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39873 } }
I20260810 15:20:48.201040 8430 leader_election.cc:302] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b0129b51e14d8e92864742e52ce477; no voters:
I20260810 15:20:48.201251 8430 leader_election.cc:288] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:48.201346 8434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:48.201589 8434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [term 1 LEADER]: Becoming Leader. State: Replica: 12b0129b51e14d8e92864742e52ce477, State: Running, Role: LEADER
I20260810 15:20:48.201679 8430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:48.202438 8434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0129b51e14d8e92864742e52ce477" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39873 } }
I20260810 15:20:48.203261 8436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b0129b51e14d8e92864742e52ce477. Latest consensus state: current_term: 1 leader_uuid: "12b0129b51e14d8e92864742e52ce477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0129b51e14d8e92864742e52ce477" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39873 } } }
I20260810 15:20:48.203375 8436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:48.203544 8437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b0129b51e14d8e92864742e52ce477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b0129b51e14d8e92864742e52ce477" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39873 } } }
I20260810 15:20:48.203681 8437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:48.204368 8450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:48.204473 8450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:48.204548 8447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:48.204753 8447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:48.206535 8447 catalog_manager.cc:1347] Generated new cluster ID: d1f511e04fb54e18849f7224e5e777c2
I20260810 15:20:48.206615 8447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:48.215643 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.201148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.4.64.126:39873)
0810 15:20:48.201373 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:48.201386 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:48.201554 (+ 168us) server_negotiation.cc:407] Connection header received
0810 15:20:48.201726 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:48.201734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:48.201793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:48.201974 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:48.203167 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.204947 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.205977 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.206791 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.206912 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.210278 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:48.210283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:48.210283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:48.210301 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:48.213406 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.213951 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.213955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.213955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.214127 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.214386 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.214387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.214388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.214856 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:48.215067 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:48.215192 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 15:20:48.215386 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260810 15:20:48.224021 8447 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:48.225102 8447 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:48.231994 8447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12b0129b51e14d8e92864742e52ce477: Generated new TSK 0
I20260810 15:20:48.232265 8447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:48.270452 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39873
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:48.359490 8456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:48.359908 8456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:48.359974 8456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:48.362218 8456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:48.362285 8456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:48.362351 8456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:48.364990 8456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:48.365509 8456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:48.365857 8456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:48.372294 8462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.372390 8461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.372303 8464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:48.372412 8456 server_base.cc:1034] running on GCE node
I20260810 15:20:48.372855 8456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:48.373142 8456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:48.374292 8456 hybrid_clock.cc:648] HybridClock initialized: now 1786375248374322 us; error 74 us; skew 500 ppm
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:48.377110 8456 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:48.377669 8456 webserver.cc:460] Webserver started at http://127.4.64.65:44931/ using document root <none> and password file <none>
I20260810 15:20:48.377935 8456 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:48.378007 8456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:48.378165 8456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:48.379300 8456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "baa51ad72e474f83905fb2a3fe18ab0f"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "6d2fbe11a21acdca858121a1a9ea42eb"
server_key_iv: "87c4de79ec088e07e5142fdcf0bc17d2"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.379685 8456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "baa51ad72e474f83905fb2a3fe18ab0f"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "6d2fbe11a21acdca858121a1a9ea42eb"
server_key_iv: "87c4de79ec088e07e5142fdcf0bc17d2"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.381134 8456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:48.382241 8472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.382498 8456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:48.382624 8456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "baa51ad72e474f83905fb2a3fe18ab0f"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "6d2fbe11a21acdca858121a1a9ea42eb"
server_key_iv: "87c4de79ec088e07e5142fdcf0bc17d2"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.382740 8456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:48.397796 8456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:48.400437 8456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:48.400665 8456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:48.400946 8456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:48.401293 8456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:48.401350 8456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.401393 8456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:48.401440 8456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.408365 8456 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:39127
I20260810 15:20:48.408427 8602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:39127 every 8 connection(s)
I20260810 15:20:48.408818 8456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:48.418016 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8456
I20260810 15:20:48.418141 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:48.418484 4353 external_mini_cluster.cc:1428] Setting key 4705943b8830e7e0afab0b8b83c068c1
I20260810 15:20:48.421482 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.409924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:50045 (local address 127.4.64.126:39873)
0810 15:20:48.410015 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:48.410017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:48.410320 (+ 303us) server_negotiation.cc:407] Connection header received
0810 15:20:48.411443 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:48.411445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:48.411464 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:48.411520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:48.412309 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.412895 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.414212 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.414457 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.414565 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.417432 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:48.417433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:48.417434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:48.417446 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:48.419470 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.420209 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.420210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.420211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.420281 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.420690 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.420691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.420691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.420838 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:48.420908 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:48.421320 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 15:20:48.421413 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:20:48.421871 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.409885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:39873 (local address 127.4.64.65:50045)
0810 15:20:48.410097 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:48.410141 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:48.411295 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:48.411622 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:48.411626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:48.411956 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:48.412193 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:48.412197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.413021 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.413023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:48.414091 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:48.414095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.414582 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.414584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:48.414759 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.415515 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:48.415526 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:48.417301 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:48.419586 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:48.419595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:48.419598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:48.420083 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:48.420374 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:48.420375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:48.420383 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:48.420585 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:48.420922 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:48.420930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:48.421141 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 15:20:48.421287 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 15:20:48.421512 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260810 15:20:48.422636 8603 heartbeater.cc:347] Connected to a master server at 127.4.64.126:39873
I20260810 15:20:48.422798 8603 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:48.423040 8603 heartbeater.cc:511] Master 127.4.64.126:39873 requested a full tablet report, sending...
I20260810 15:20:48.423682 8389 ts_manager.cc:194] Registered new tserver with Master: baa51ad72e474f83905fb2a3fe18ab0f (127.4.64.65:39127)
I20260810 15:20:48.424297 8389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:50045
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:48.454252 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39873
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:48.541769 8608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:48.541998 8608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:48.542026 8608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:48.543222 8608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:48.543268 8608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:48.543295 8608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:48.544821 8608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:48.545060 8608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:48.545279 8608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:48.551707 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.551739 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.551718 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:48.552093 8608 server_base.cc:1034] running on GCE node
I20260810 15:20:48.552371 8608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:48.552673 8608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:48.553893 8608 hybrid_clock.cc:648] HybridClock initialized: now 1786375248553874 us; error 30 us; skew 500 ppm
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:48.556919 8608 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:48.557411 8608 webserver.cc:460] Webserver started at http://127.4.64.66:45701/ using document root <none> and password file <none>
I20260810 15:20:48.557699 8608 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:48.557755 8608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:48.557857 8608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:48.558926 8608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "54e0f501a17546f7ad1a05579411b7aa"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "57b93b7a1d7b80ef55653227560706b4"
server_key_iv: "3688f1a1cf6013cc9219860c6d4e844a"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.559264 8608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "54e0f501a17546f7ad1a05579411b7aa"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "57b93b7a1d7b80ef55653227560706b4"
server_key_iv: "3688f1a1cf6013cc9219860c6d4e844a"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.560820 8608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:48.561749 8624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.561964 8608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:48.562047 8608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "54e0f501a17546f7ad1a05579411b7aa"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "57b93b7a1d7b80ef55653227560706b4"
server_key_iv: "3688f1a1cf6013cc9219860c6d4e844a"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.562151 8608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:48.603860 8608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:48.606503 8608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:48.606668 8608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:48.606993 8608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:48.607424 8608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:48.607473 8608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.607506 8608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:48.607527 8608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.614799 8608 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:35567
I20260810 15:20:48.614845 8753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:35567 every 8 connection(s)
I20260810 15:20:48.615212 8608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:48.621933 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8608
I20260810 15:20:48.622074 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:48.622313 4353 external_mini_cluster.cc:1428] Setting key 7d9311503751aac57f4f180d7c2d2c9e
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:48.627189 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.616052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:40717 (local address 127.4.64.126:39873)
0810 15:20:48.616156 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:48.616158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:48.616419 (+ 261us) server_negotiation.cc:407] Connection header received
0810 15:20:48.617481 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:48.617483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:48.617495 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:48.617549 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:48.618426 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.618976 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.620295 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.620523 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.620598 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.623469 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:48.623471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:48.623471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:48.623480 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:48.625265 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.625932 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.625934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.625934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.625998 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.626395 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.626397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.626397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.626543 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:48.626609 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:48.626997 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 15:20:48.627100 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 15:20:48.627461 8756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.615988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:39873 (local address 127.4.64.66:40717)
0810 15:20:48.616258 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:48.616305 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:48.617372 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:48.617670 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:48.617675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:48.618078 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:48.618321 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:48.618327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.619138 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.619139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:48.620157 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:48.620160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.620639 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.620640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:48.620783 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.621362 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:48.621372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:48.623363 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:48.625383 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:48.625395 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:48.625398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:48.625817 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:48.626073 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:48.626074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:48.626074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:48.626274 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:48.626621 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:48.626628 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:48.626825 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 15:20:48.626940 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:20:48.627158 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260810 15:20:48.628098 8754 heartbeater.cc:347] Connected to a master server at 127.4.64.126:39873
I20260810 15:20:48.628237 8754 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:48.628479 8754 heartbeater.cc:511] Master 127.4.64.126:39873 requested a full tablet report, sending...
I20260810 15:20:48.628981 8389 ts_manager.cc:194] Registered new tserver with Master: 54e0f501a17546f7ad1a05579411b7aa (127.4.64.66:35567)
I20260810 15:20:48.629444 8389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:40717
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:48.658344 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39873
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:48.748514 8760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:48.748780 8760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:48.748824 8760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:48.750108 8760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:48.750190 8760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:48.750236 8760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:48.751917 8760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:48.752250 8760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:48.752521 8760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:48.758020 8766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.758025 8768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:48.758025 8765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:48.758047 8760 server_base.cc:1034] running on GCE node
I20260810 15:20:48.758594 8760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:48.758912 8760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:48.760130 8760 hybrid_clock.cc:648] HybridClock initialized: now 1786375248760073 us; error 73 us; skew 500 ppm
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:48.763231 8760 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:48.763886 8760 webserver.cc:460] Webserver started at http://127.4.64.67:34243/ using document root <none> and password file <none>
I20260810 15:20:48.764181 8760 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:48.764261 8760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:48.764444 8760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:48.765847 8760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "495a92db5e164e12a0af9cfa836b9f28"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "6f95a279a879381e7c15fcb065fa81f7"
server_key_iv: "79d914661cba48a1e3a958f8089b2824"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.766258 8760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "495a92db5e164e12a0af9cfa836b9f28"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "6f95a279a879381e7c15fcb065fa81f7"
server_key_iv: "79d914661cba48a1e3a958f8089b2824"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.767928 8760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:48.768918 8775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.769143 8760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:48.769325 8760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "495a92db5e164e12a0af9cfa836b9f28"
format_stamp: "Formatted at 2026-08-10 15:20:48 on dist-test-slave-9zdj"
server_key: "6f95a279a879381e7c15fcb065fa81f7"
server_key_iv: "79d914661cba48a1e3a958f8089b2824"
server_key_version: "encryptionkey@0"
I20260810 15:20:48.769425 8760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:48.807205 8760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:48.810160 8760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:48.810343 8760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:48.810626 8760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:48.811012 8760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:48.811075 8760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.811129 8760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:48.811158 8760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:48.817760 8760 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:33847
I20260810 15:20:48.817831 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:33847 every 8 connection(s)
I20260810 15:20:48.818243 8760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:48.825791 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8760
I20260810 15:20:48.825892 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:48.826093 4353 external_mini_cluster.cc:1428] Setting key 45bf885382531234563fd69a4fd0abdd
I20260810 15:20:48.829607 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.819279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:37025 (local address 127.4.64.126:39873)
0810 15:20:48.819421 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:48.819424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:48.819606 (+ 182us) server_negotiation.cc:407] Connection header received
0810 15:20:48.820712 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:48.820714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:48.820730 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:48.820786 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:48.821499 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.822101 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.823192 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.823409 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.823503 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.826316 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:48.826318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:48.826318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:48.826327 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:48.827972 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.828517 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.828518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.828518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.828579 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.828833 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.828834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.828834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.828957 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:48.829017 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:48.829459 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 15:20:48.829537 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 15:20:48.829949 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.819180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:39873 (local address 127.4.64.67:37025)
0810 15:20:48.819461 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:48.819515 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:48.820560 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:48.820865 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:48.820869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:48.821166 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:48.821379 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:48.821385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.822225 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.822226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:48.823087 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:48.823089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.823519 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.823520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:48.823654 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.824392 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:48.824402 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:48.826204 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:48.828058 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:48.828065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:48.828067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:48.828437 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:48.828651 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:48.828652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:48.828652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:48.828767 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:48.829058 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:48.829065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:48.829286 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 15:20:48.829408 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 15:20:48.829564 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260810 15:20:48.830499 8902 heartbeater.cc:347] Connected to a master server at 127.4.64.126:39873
I20260810 15:20:48.830637 8902 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:48.830897 8902 heartbeater.cc:511] Master 127.4.64.126:39873 requested a full tablet report, sending...
I20260810 15:20:48.831528 8389 ts_manager.cc:194] Registered new tserver with Master: 495a92db5e164e12a0af9cfa836b9f28 (127.4.64.67:33847)
I20260810 15:20:48.831905 8389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:37025
I20260810 15:20:48.839056 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:48.867054 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.857476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.4.64.65:39127)
0810 15:20:48.857756 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:48.857762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:48.857774 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:48.857802 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:48.857804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:48.857819 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:48.857892 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:48.858556 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.859862 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.860676 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.861051 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.861115 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.863856 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:48.863869 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:48.863870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:48.863891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:48.865859 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.866213 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.866217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.866217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.866278 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.866449 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.866450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.866450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.866601 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:48.866714 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:48.866861 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:20:48.866936 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":169,"threads_started":1}
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:48.877266 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:48.868824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.4.64.126:39873)
0810 15:20:48.868918 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:48.868920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:48.868928 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:48.868952 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:48.868952 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:48.868962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:48.869010 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:48.869528 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.870129 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.870825 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:48.871030 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:48.871081 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:48.874106 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:48.874108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:48.874109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:48.874119 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:48.875824 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.876300 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.876301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.876301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.876371 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:48.876611 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:48.876612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:48.876612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:48.876804 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:48.876928 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:48.877052 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 15:20:48.877149 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 15:20:48.877578 8389 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49900
I20260810 15:20:48.878001 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8456
I20260810 15:20:48.891575 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8608
I20260810 15:20:48.901491 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8760
I20260810 15:20:48.912707 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8353
2026-08-10T15:20:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2220 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8919](info): set up 2 sockets
Aug 10 15:20:48 dist-test-slave-9zdj krb5kdc[8919](info): commencing operation
krb5kdc: starting...
W20260810 15:20:50.135958 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:50.282749 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:46283
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:50.371703 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:50.371918 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:50.371940 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:50.373181 8933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:50.373232 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:50.373247 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:50.373260 8933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:50.373272 8933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:50.374881 8933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:46283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:50.375183 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:50.375442 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:50.380743 8940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:50.380759 8942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:50.380790 8933 server_base.cc:1034] running on GCE node
W20260810 15:20:50.380759 8939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:50.381278 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:50.381603 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:50.382838 8933 hybrid_clock.cc:648] HybridClock initialized: now 1786375250382820 us; error 44 us; skew 500 ppm
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:50.385761 8933 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:50.386101 8933 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:50.386157 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:50.386252 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:50.387447 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "a27f0ea67d394f3b950e74c0ce2fafc4"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "b48b49d45cadbf4b83de89a651651606"
server_key_iv: "e2b37e1a894a7ea37cdce15bb5ad8c5b"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.387769 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "a27f0ea67d394f3b950e74c0ce2fafc4"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "b48b49d45cadbf4b83de89a651651606"
server_key_iv: "e2b37e1a894a7ea37cdce15bb5ad8c5b"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.389156 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:20:50.390092 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.390280 8933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:50.390384 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "a27f0ea67d394f3b950e74c0ce2fafc4"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "b48b49d45cadbf4b83de89a651651606"
server_key_iv: "e2b37e1a894a7ea37cdce15bb5ad8c5b"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.390450 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:50.398532 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:50.400465 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:50.400578 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:50.404731 8933 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:46283
I20260810 15:20:50.404801 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:46283 every 8 connection(s)
I20260810 15:20:50.405134 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:50.406466 9010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:50.408910 9010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4: Bootstrap starting.
I20260810 15:20:50.409477 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 8933
I20260810 15:20:50.409590 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:50.409750 9010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:50.409835 4353 external_mini_cluster.cc:1428] Setting key 9ea163fe76879561a9f4a38c7b4f3c2c
I20260810 15:20:50.410115 9010 log.cc:824] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:50.410929 9010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4: No bootstrap required, opened a new log
I20260810 15:20:50.413498 9010 raft_consensus.cc:348] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46283 } }
I20260810 15:20:50.413733 9010 raft_consensus.cc:374] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:50.413775 9010 raft_consensus.cc:729] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27f0ea67d394f3b950e74c0ce2fafc4, State: Initialized, Role: FOLLOWER
I20260810 15:20:50.413903 9010 consensus_queue.cc:260] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46283 } }
I20260810 15:20:50.413978 9010 raft_consensus.cc:388] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:50.414006 9010 raft_consensus.cc:482] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:50.414042 9010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:50.414788 9010 raft_consensus.cc:504] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46283 } }
I20260810 15:20:50.414884 9010 leader_election.cc:302] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27f0ea67d394f3b950e74c0ce2fafc4; no voters:
I20260810 15:20:50.415078 9010 leader_election.cc:288] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:50.415233 9015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:50.415405 9015 raft_consensus.cc:686] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [term 1 LEADER]: Becoming Leader. State: Replica: a27f0ea67d394f3b950e74c0ce2fafc4, State: Running, Role: LEADER
I20260810 15:20:50.415524 9010 sys_catalog.cc:564] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:50.415551 9015 consensus_queue.cc:237] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46283 } }
I20260810 15:20:50.416867 9016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a27f0ea67d394f3b950e74c0ce2fafc4. Latest consensus state: current_term: 1 leader_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46283 } } }
I20260810 15:20:50.416960 9016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [sys.catalog]: This master's current role is: LEADER
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:50.417716 9018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27f0ea67d394f3b950e74c0ce2fafc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46283 } } }
I20260810 15:20:50.417814 9018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:50.417851 9033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:50.418484 9033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:50.420425 9033 catalog_manager.cc:1347] Generated new cluster ID: 06557e03fe684002aac6985d35c5a9d0
I20260810 15:20:50.420488 9033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:50.422353 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.410632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.4.64.126:46283)
0810 15:20:50.410815 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:50.410825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:50.410859 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:20:50.411338 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:50.411345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:50.411396 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:50.411515 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:50.412339 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.413235 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.414408 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.414988 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.415135 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.418148 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:50.418152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:50.418152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:50.418164 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:50.420159 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.420694 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.420698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.420699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.420799 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.420986 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.420987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.420987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.421286 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:50.421427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:50.421975 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 15:20:50.422117 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260810 15:20:50.428778 9033 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:50.429556 9033 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:50.433811 9033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4: Generated new TSK 0
I20260810 15:20:50.434028 9033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:50.477507 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46283
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:50.576654 9038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:50.576939 9038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:50.576987 9038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:50.578279 9038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:50.578352 9038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:50.578399 9038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:50.580009 9038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:50.580295 9038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:50.580542 9038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:50.585971 9044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:50.585971 9046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:50.586066 9043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:50.586117 9038 server_base.cc:1034] running on GCE node
I20260810 15:20:50.586500 9038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:50.586723 9038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:50.587862 9038 hybrid_clock.cc:648] HybridClock initialized: now 1786375250587835 us; error 45 us; skew 500 ppm
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:50.590720 9038 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:50.591115 9038 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:50.591189 9038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:50.591357 9038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:50.592499 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "1eeec2d61942436993837416b4ff75ad"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "a7d9b184befbedb43d3ec0d323a439b9"
server_key_iv: "f635e85971c980e86e7793c2288cbd0c"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.592880 9038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "1eeec2d61942436993837416b4ff75ad"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "a7d9b184befbedb43d3ec0d323a439b9"
server_key_iv: "f635e85971c980e86e7793c2288cbd0c"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.594508 9038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:50.595369 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.595562 9038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:50.595667 9038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "1eeec2d61942436993837416b4ff75ad"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "a7d9b184befbedb43d3ec0d323a439b9"
server_key_iv: "f635e85971c980e86e7793c2288cbd0c"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.595813 9038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:50.608928 9038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:50.611186 9038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:50.611423 9038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:50.611691 9038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:50.612030 9038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:50.612088 9038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.612131 9038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:50.612164 9038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.619616 9038 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:46013
I20260810 15:20:50.619652 9179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:46013 every 8 connection(s)
I20260810 15:20:50.619992 9038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:50.625008 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9038
I20260810 15:20:50.625177 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:50.625422 4353 external_mini_cluster.cc:1428] Setting key 8df39bae94d1c79e1714eaf9098e1393
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:50.630566 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.620866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:38331 (local address 127.4.64.126:46283)
0810 15:20:50.620937 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:50.620939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:50.621129 (+ 190us) server_negotiation.cc:407] Connection header received
0810 15:20:50.621719 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:50.621721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:50.621736 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:50.621818 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:50.622806 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.623331 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.624406 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.624611 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.624671 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.627158 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:50.627159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:50.627160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:50.627169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:50.628836 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.629411 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.629412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.629413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.629474 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.629834 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.629835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.629835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.629976 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:50.630029 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:50.630407 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 15:20:50.630495 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:20:50.630877 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.620813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:46283 (local address 127.4.64.65:38331)
0810 15:20:50.620996 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:50.621042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:50.621549 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:50.621900 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:50.621904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:50.622191 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:50.622709 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:50.622716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.623419 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.623419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:50.624314 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:50.624317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.624691 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.624692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:50.624804 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.625455 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:50.625468 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:50.627037 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:50.628928 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:50.628938 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:50.628941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:50.629299 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:50.629546 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:50.629547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:50.629548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:50.629752 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:50.630041 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:50.630049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:50.630242 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 15:20:50.630365 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 15:20:50.630567 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260810 15:20:50.631604 9180 heartbeater.cc:347] Connected to a master server at 127.4.64.126:46283
I20260810 15:20:50.631750 9180 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:50.631968 9180 heartbeater.cc:511] Master 127.4.64.126:46283 requested a full tablet report, sending...
I20260810 15:20:50.632594 8968 ts_manager.cc:194] Registered new tserver with Master: 1eeec2d61942436993837416b4ff75ad (127.4.64.65:46013)
I20260810 15:20:50.633277 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:38331
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:50.658022 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46283
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:50.745028 9187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:50.745283 9187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:50.745311 9187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:50.746572 9187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:50.746622 9187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:50.746651 9187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:50.748204 9187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:50.748479 9187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:50.748752 9187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:50.754361 9187 server_base.cc:1034] running on GCE node
W20260810 15:20:50.754300 9192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:50.754349 9193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:50.754292 9195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:50.754839 9187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:50.755082 9187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:50.756235 9187 hybrid_clock.cc:648] HybridClock initialized: now 1786375250756206 us; error 44 us; skew 500 ppm
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:50.759089 9187 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:50.759536 9187 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:50.759615 9187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:50.759774 9187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:50.760885 9187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "3485c43b33b4413ba8d60151624a3cc5"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "570ab839cc8980de33cefc07d60ce01c"
server_key_iv: "e650bb7db5fce1e24bce75dfe8491aac"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.761268 9187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "3485c43b33b4413ba8d60151624a3cc5"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "570ab839cc8980de33cefc07d60ce01c"
server_key_iv: "e650bb7db5fce1e24bce75dfe8491aac"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.762712 9187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:50.763769 9203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.764038 9187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:50.764174 9187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "3485c43b33b4413ba8d60151624a3cc5"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "570ab839cc8980de33cefc07d60ce01c"
server_key_iv: "e650bb7db5fce1e24bce75dfe8491aac"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.764256 9187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:50.793107 9187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:50.795460 9187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:50.795622 9187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:50.795904 9187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:50.796257 9187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:50.796314 9187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.796365 9187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:50.796392 9187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.803129 9187 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:46319
I20260810 15:20:50.803171 9332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:46319 every 8 connection(s)
I20260810 15:20:50.803508 9187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:50.804576 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9187
I20260810 15:20:50.804701 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:50.804930 4353 external_mini_cluster.cc:1428] Setting key 7d209213e6a3aaf419e4d62dfc26ca36
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:50.816183 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.804337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:47895 (local address 127.4.64.126:46283)
0810 15:20:50.804411 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:50.804413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:50.805343 (+ 930us) server_negotiation.cc:407] Connection header received
0810 15:20:50.806045 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:50.806046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:50.806057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:50.806108 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:50.807511 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.808071 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.809185 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.809485 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.809559 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.812772 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:50.812774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:50.812774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:50.812782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:50.814307 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.814917 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.814919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.814919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.814993 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.815310 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.815311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.815311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.815457 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:50.815533 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:50.816020 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 15:20:50.816111 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
I20260810 15:20:50.816545 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.804260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:46283 (local address 127.4.64.66:47895)
0810 15:20:50.805182 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:50.805223 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:50.805922 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:50.806192 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:50.806196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:50.806625 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:50.807393 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:50.807400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.808189 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.808190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:50.809078 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:50.809080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.809579 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.809580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:50.809789 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.810405 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:50.810415 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:50.812652 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:50.814414 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:50.814424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:50.814428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:50.814810 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:50.815066 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:50.815067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:50.815068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:50.815234 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:50.815544 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:50.815553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:50.815819 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 15:20:50.815964 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 15:20:50.816254 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":102,"threads_started":1}
I20260810 15:20:50.817202 9333 heartbeater.cc:347] Connected to a master server at 127.4.64.126:46283
I20260810 15:20:50.817338 9333 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:50.817528 9333 heartbeater.cc:511] Master 127.4.64.126:46283 requested a full tablet report, sending...
I20260810 15:20:50.818069 8968 ts_manager.cc:194] Registered new tserver with Master: 3485c43b33b4413ba8d60151624a3cc5 (127.4.64.66:46319)
I20260810 15:20:50.818497 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:47895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:50.837334 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46283
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:50.931207 9338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:50.931507 9338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:50.931555 9338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:50.932942 9338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:50.933034 9338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:50.933085 9338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:50.934862 9338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:50.935182 9338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:50.935460 9338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:50.941097 9343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:50.941114 9338 server_base.cc:1034] running on GCE node
W20260810 15:20:50.941126 9344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:50.941097 9346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:50.941702 9338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:50.941947 9338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:50.943157 9338 hybrid_clock.cc:648] HybridClock initialized: now 1786375250943138 us; error 33 us; skew 500 ppm
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:50.946417 9338 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:50.946817 9338 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:50.946871 9338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:50.947023 9338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:50.948144 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "240a37b69cc94062a8c1f4fc92829628"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "eb18ffd87980838b8ea25f2fb73369eb"
server_key_iv: "78b8120b8190346c601061f255135ee6"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.948534 9338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "240a37b69cc94062a8c1f4fc92829628"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "eb18ffd87980838b8ea25f2fb73369eb"
server_key_iv: "78b8120b8190346c601061f255135ee6"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.950019 9338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:50.950904 9353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.951094 9338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:50.951264 9338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "240a37b69cc94062a8c1f4fc92829628"
format_stamp: "Formatted at 2026-08-10 15:20:50 on dist-test-slave-9zdj"
server_key: "eb18ffd87980838b8ea25f2fb73369eb"
server_key_iv: "78b8120b8190346c601061f255135ee6"
server_key_version: "encryptionkey@0"
I20260810 15:20:50.951355 9338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:50.969192 9338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:50.971799 9338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:50.971982 9338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:50.972252 9338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:50.972595 9338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:50.972651 9338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.972694 9338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:50.972718 9338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:50.979101 9338 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:38967
I20260810 15:20:50.979182 9488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:38967 every 8 connection(s)
I20260810 15:20:50.979483 9338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:50.984061 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9338
I20260810 15:20:50.984187 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:50.984400 4353 external_mini_cluster.cc:1428] Setting key c132d5f253aaa9a1a48875059d1943c1
Aug 10 15:20:50 dist-test-slave-9zdj krb5kdc[8919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375250, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:50.990267 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.980430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:41953 (local address 127.4.64.126:46283)
0810 15:20:50.980558 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:50.980560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:50.980766 (+ 206us) server_negotiation.cc:407] Connection header received
0810 15:20:50.981416 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:50.981417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:50.981429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:50.981477 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:50.982605 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.983133 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.984264 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.984573 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.984652 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.987255 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:50.987256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:50.987256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:50.987263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:50.988938 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.989320 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.989321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.989322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.989376 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:50.989602 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:50.989603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:50.989603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:50.989794 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:50.989851 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:50.990125 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 15:20:50.990197 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 15:20:50.990494 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.980377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:46283 (local address 127.4.64.67:41953)
0810 15:20:50.980622 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:50.980663 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:50.981297 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:50.981542 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:50.981549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:50.981917 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:50.982478 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:50.982484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.983241 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.983242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:50.984131 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:50.984134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.984692 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.984694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:50.984866 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:50.985657 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:50.985667 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:50.987156 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:50.989015 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:50.989022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:50.989024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:50.989251 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:50.989443 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:50.989443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:50.989443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:50.989546 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:50.989860 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:50.989867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:50.990003 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 15:20:50.990097 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 15:20:50.990230 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260810 15:20:50.991065 9489 heartbeater.cc:347] Connected to a master server at 127.4.64.126:46283
I20260810 15:20:50.991173 9489 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:50.991333 9489 heartbeater.cc:511] Master 127.4.64.126:46283 requested a full tablet report, sending...
I20260810 15:20:50.991747 8968 ts_manager.cc:194] Registered new tserver with Master: 240a37b69cc94062a8c1f4fc92829628 (127.4.64.67:38967)
I20260810 15:20:50.992092 8968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:41953
I20260810 15:20:50.996300 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:20:51.003916 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:50.997675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.4.64.126:46283)
0810 15:20:50.997741 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:50.997744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:50.997844 (+ 100us) server_negotiation.cc:407] Connection header received
0810 15:20:50.997887 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:50.997888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:50.997896 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:50.997942 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:50.998543 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:50.999042 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:50.999830 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.000010 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.000083 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.001320 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.001322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.001322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:51.001329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.002905 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:51.003213 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:51.003214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:51.003215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:51.003260 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:51.003411 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:51.003411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:51.003411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:51.003548 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.003625 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.003727 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.003793 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:20:51.005920 8968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:51.007483 8968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:51.014504 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.010051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46013 (local address 127.0.0.1:46566)
0810 15:20:51.010381 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.010429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.010502 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.010827 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.010830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.010832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:51.010914 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.010917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.012003 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.012004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.013137 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.013140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.014144 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.014145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.014238 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.014254 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.014342 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.014409 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":66,"threads_started":1}
I20260810 15:20:51.014521 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.010496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.4.64.65:46013)
0810 15:20:51.010641 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.010646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.010657 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:51.010703 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.010705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.010723 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.010821 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:51.011000 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.011903 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.013248 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.014046 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.014128 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.014202 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.014364 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.014419 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260810 15:20:51.014838 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.010404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.4.64.66:46319)
0810 15:20:51.010578 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.010582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.011064 (+ 482us) server_negotiation.cc:407] Connection header received
0810 15:20:51.011156 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.011158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.011185 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.011264 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:51.011407 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.012437 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.013595 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.014157 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.014236 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.014296 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.014547 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.014751 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260810 15:20:51.014837 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.010051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:38967 (local address 127.0.0.1:34524)
0810 15:20:51.010606 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.010614 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.010675 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.010995 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.010996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.010998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:51.011047 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.011048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.012411 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.012413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.013503 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.013506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.014533 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.014534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.014619 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.014623 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.014719 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.014766 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260810 15:20:51.014865 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.010256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.4.64.67:38967)
0810 15:20:51.010461 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.010465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.010671 (+ 206us) server_negotiation.cc:407] Connection header received
0810 15:20:51.010779 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.010782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.010798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.010881 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:51.011134 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.012286 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.013604 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.014370 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.014415 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.014465 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.014745 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.014789 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":110,"threads_started":1}
I20260810 15:20:51.015372 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.010074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:46319 (local address 127.0.0.1:43078)
0810 15:20:51.010998 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.011010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.011079 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.011274 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.011276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.011278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:51.011343 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.011344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.012541 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.012542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.013505 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.013507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.014332 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.014333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.014399 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.014402 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.014449 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.014483 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":108,"threads_started":1}
I20260810 15:20:51.015767 9410 tablet_service.cc:1467] Processing CreateTablet for tablet 4a99d6b0ae7b4b1e8e91a30293303be0 (DEFAULT_TABLE table=test-table [id=1503d6d834284e50b81b73156eda4ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:51.015810 9106 tablet_service.cc:1467] Processing CreateTablet for tablet 4a99d6b0ae7b4b1e8e91a30293303be0 (DEFAULT_TABLE table=test-table [id=1503d6d834284e50b81b73156eda4ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:51.016072 9255 tablet_service.cc:1467] Processing CreateTablet for tablet 4a99d6b0ae7b4b1e8e91a30293303be0 (DEFAULT_TABLE table=test-table [id=1503d6d834284e50b81b73156eda4ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:51.016216 9410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a99d6b0ae7b4b1e8e91a30293303be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.016225 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a99d6b0ae7b4b1e8e91a30293303be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.016381 9255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a99d6b0ae7b4b1e8e91a30293303be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.019341 9514 tablet_bootstrap.cc:492] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: Bootstrap starting.
I20260810 15:20:51.019668 9515 tablet_bootstrap.cc:492] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Bootstrap starting.
I20260810 15:20:51.020299 9515 tablet_bootstrap.cc:654] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.020299 9514 tablet_bootstrap.cc:654] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.020989 9515 log.cc:824] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.020989 9514 log.cc:824] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.021363 9516 tablet_bootstrap.cc:492] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Bootstrap starting.
I20260810 15:20:51.022006 9515 tablet_bootstrap.cc:492] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: No bootstrap required, opened a new log
I20260810 15:20:51.022032 9516 tablet_bootstrap.cc:654] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.022230 9514 tablet_bootstrap.cc:492] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: No bootstrap required, opened a new log
I20260810 15:20:51.022111 9515 ts_tablet_manager.cc:1397] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:51.022308 9514 ts_tablet_manager.cc:1397] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:51.022624 9516 log.cc:824] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.023484 9516 tablet_bootstrap.cc:492] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: No bootstrap required, opened a new log
I20260810 15:20:51.023550 9516 ts_tablet_manager.cc:1397] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:20:51.024384 9515 raft_consensus.cc:348] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.024456 9514 raft_consensus.cc:348] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.024566 9515 raft_consensus.cc:374] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.024616 9515 raft_consensus.cc:729] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3485c43b33b4413ba8d60151624a3cc5, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.024617 9514 raft_consensus.cc:374] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.024653 9514 raft_consensus.cc:729] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eeec2d61942436993837416b4ff75ad, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.024770 9515 consensus_queue.cc:260] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.024784 9514 consensus_queue.cc:260] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.025095 9514 ts_tablet_manager.cc:1428] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:51.025112 9333 heartbeater.cc:503] Master 127.4.64.126:46283 was elected leader, sending a full tablet report...
I20260810 15:20:51.025233 9180 heartbeater.cc:503] Master 127.4.64.126:46283 was elected leader, sending a full tablet report...
I20260810 15:20:51.025655 9516 raft_consensus.cc:348] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.025810 9516 raft_consensus.cc:374] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.025862 9516 raft_consensus.cc:729] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240a37b69cc94062a8c1f4fc92829628, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.025861 9515 ts_tablet_manager.cc:1428] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 15:20:51.026007 9516 consensus_queue.cc:260] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.026309 9516 ts_tablet_manager.cc:1428] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 15:20:51.026331 9489 heartbeater.cc:503] Master 127.4.64.126:46283 was elected leader, sending a full tablet report...
W20260810 15:20:51.054105 9334 tablet.cc:2369] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:51.092234 9520 raft_consensus.cc:482] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:51.092586 9520 raft_consensus.cc:504] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.093067 9520 leader_election.cc:288] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 240a37b69cc94062a8c1f4fc92829628 (127.4.64.67:38967), 3485c43b33b4413ba8d60151624a3cc5 (127.4.64.66:46319)
I20260810 15:20:51.097301 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.093397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:60413 (local address 127.4.64.66:46319)
0810 15:20:51.093447 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.093450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.093461 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:51.093495 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.093496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.093508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.093585 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:51.093791 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.094614 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.096301 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.096927 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.096993 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.097003 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.097178 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.097226 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:20:51.097379 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.093419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:49373 (local address 127.4.64.67:38967)
0810 15:20:51.093465 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.093467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.093667 (+ 200us) server_negotiation.cc:407] Connection header received
0810 15:20:51.093792 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.093793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.093807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.093869 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:51.094015 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.094686 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.096311 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.096899 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.096975 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.096989 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.097261 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.097306 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:51.097311 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.093279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:46319 (local address 127.4.64.65:60413)
0810 15:20:51.093355 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.093368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.093418 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.093641 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.093642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.093644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:51.093710 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.093712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.094750 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.094751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.096120 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.096124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.097005 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.097007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.097091 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.097095 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.097152 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.097196 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":98}
I20260810 15:20:51.097317 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.093279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:38967 (local address 127.4.64.65:49373)
0810 15:20:51.093547 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.093564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.093702 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.093885 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.093886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.093887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:51.093940 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.093941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.094764 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.094765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.096120 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:51.096124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.097005 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.097007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:51.097090 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.097095 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.097152 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.097196 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":87,"thread_start_us":135,"threads_started":1}
I20260810 15:20:51.097983 9435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a99d6b0ae7b4b1e8e91a30293303be0" candidate_uuid: "1eeec2d61942436993837416b4ff75ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240a37b69cc94062a8c1f4fc92829628" is_pre_election: true
I20260810 15:20:51.098067 9279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a99d6b0ae7b4b1e8e91a30293303be0" candidate_uuid: "1eeec2d61942436993837416b4ff75ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3485c43b33b4413ba8d60151624a3cc5" is_pre_election: true
I20260810 15:20:51.098155 9435 raft_consensus.cc:2454] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eeec2d61942436993837416b4ff75ad in term 0.
I20260810 15:20:51.098268 9279 raft_consensus.cc:2454] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eeec2d61942436993837416b4ff75ad in term 0.
I20260810 15:20:51.098472 9059 leader_election.cc:302] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeec2d61942436993837416b4ff75ad, 240a37b69cc94062a8c1f4fc92829628; no voters:
I20260810 15:20:51.098672 9520 raft_consensus.cc:2781] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:51.098713 9520 raft_consensus.cc:482] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:51.098729 9520 raft_consensus.cc:3037] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:51.099422 9520 raft_consensus.cc:504] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.099577 9520 leader_election.cc:288] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [CANDIDATE]: Term 1 election: Requested vote from peers 240a37b69cc94062a8c1f4fc92829628 (127.4.64.67:38967), 3485c43b33b4413ba8d60151624a3cc5 (127.4.64.66:46319)
I20260810 15:20:51.099785 9435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a99d6b0ae7b4b1e8e91a30293303be0" candidate_uuid: "1eeec2d61942436993837416b4ff75ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240a37b69cc94062a8c1f4fc92829628"
I20260810 15:20:51.099895 9435 raft_consensus.cc:3037] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:51.100577 9435 raft_consensus.cc:2454] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eeec2d61942436993837416b4ff75ad in term 1.
I20260810 15:20:51.100760 9279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a99d6b0ae7b4b1e8e91a30293303be0" candidate_uuid: "1eeec2d61942436993837416b4ff75ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3485c43b33b4413ba8d60151624a3cc5"
I20260810 15:20:51.100842 9059 leader_election.cc:302] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeec2d61942436993837416b4ff75ad, 240a37b69cc94062a8c1f4fc92829628; no voters:
I20260810 15:20:51.100876 9279 raft_consensus.cc:3037] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:51.100955 9520 raft_consensus.cc:2781] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:51.101096 9520 raft_consensus.cc:686] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 1 LEADER]: Becoming Leader. State: Replica: 1eeec2d61942436993837416b4ff75ad, State: Running, Role: LEADER
I20260810 15:20:51.101315 9520 consensus_queue.cc:237] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } }
I20260810 15:20:51.101950 9279 raft_consensus.cc:2454] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eeec2d61942436993837416b4ff75ad in term 1.
I20260810 15:20:51.102326 8968 catalog_manager.cc:5495] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eeec2d61942436993837416b4ff75ad (127.4.64.65). New cstate: current_term: 1 leader_uuid: "1eeec2d61942436993837416b4ff75ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eeec2d61942436993837416b4ff75ad" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 } health_report { overall_health: UNKNOWN } } }
I20260810 15:20:51.111685 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.108881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.4.64.65:46013)
0810 15:20:51.108947 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.108949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.108959 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:51.109067 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.109068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.109090 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.109182 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:51.109346 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.109904 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.110816 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:51.111017 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:51.111100 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:51.111297 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:51.111410 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:51.111513 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.111532 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.111557 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:20:51.113783 9435 raft_consensus.cc:1261] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 1 FOLLOWER]: Refusing update from remote peer 1eeec2d61942436993837416b4ff75ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:51.114131 9279 raft_consensus.cc:1261] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 1 FOLLOWER]: Refusing update from remote peer 1eeec2d61942436993837416b4ff75ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:51.114315 9524 consensus_queue.cc:1035] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "240a37b69cc94062a8c1f4fc92829628" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 38967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:51.114482 9520 consensus_queue.cc:1035] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3485c43b33b4413ba8d60151624a3cc5" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:51.117455 9528 mvcc.cc:205] Tried to move back new op lower bound from 7316993028558884864 to 7316993028511703040. Current Snapshot: MvccSnapshot[applied={T|T < 7316993028558884864}]
I20260810 15:20:51.118158 9529 mvcc.cc:205] Tried to move back new op lower bound from 7316993028558884864 to 7316993028511703040. Current Snapshot: MvccSnapshot[applied={T|T < 7316993028558884864}]
I20260810 15:20:51.120287 8968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:51.120406 8968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:51.121218 8968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a27f0ea67d394f3b950e74c0ce2fafc4: Sending DeleteTablet for 3 replicas of tablet 4a99d6b0ae7b4b1e8e91a30293303be0
I20260810 15:20:51.121644 9106 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a99d6b0ae7b4b1e8e91a30293303be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:51 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:46566
I20260810 15:20:51.121680 9255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a99d6b0ae7b4b1e8e91a30293303be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:51 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:43078
I20260810 15:20:51.121722 9410 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a99d6b0ae7b4b1e8e91a30293303be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:51 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:34524
I20260810 15:20:51.122010 9540 tablet_replica.cc:330] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad: stopping tablet replica
I20260810 15:20:51.122138 9540 raft_consensus.cc:2229] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:51.122242 9540 raft_consensus.cc:2258] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 1eeec2d61942436993837416b4ff75ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:51.122236 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9038
I20260810 15:20:51.126271 9539 tablet_replica.cc:330] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: stopping tablet replica
I20260810 15:20:51.126446 9539 raft_consensus.cc:2229] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:51.126591 9539 raft_consensus.cc:2258] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:51.127185 9539 ts_tablet_manager.cc:1905] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:51.129355 9539 ts_tablet_manager.cc:1918] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:51.129455 9539 log.cc:1196] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/4a99d6b0ae7b4b1e8e91a30293303be0
I20260810 15:20:51.129614 9541 tablet_replica.cc:330] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: stopping tablet replica
I20260810 15:20:51.129799 9541 raft_consensus.cc:2229] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:51.129875 9539 ts_tablet_manager.cc:1939] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 3485c43b33b4413ba8d60151624a3cc5: Deleting consensus metadata
I20260810 15:20:51.129966 9541 raft_consensus.cc:2258] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:51.130452 9541 ts_tablet_manager.cc:1905] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:51.130647 8955 catalog_manager.cc:4841] TS 3485c43b33b4413ba8d60151624a3cc5 (127.4.64.66:46319): tablet 4a99d6b0ae7b4b1e8e91a30293303be0 (table test-table [id=1503d6d834284e50b81b73156eda4ed9]) successfully deleted
I20260810 15:20:51.132015 9541 ts_tablet_manager.cc:1918] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:51.132122 9541 log.cc:1196] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/4a99d6b0ae7b4b1e8e91a30293303be0
W20260810 15:20:51.132289 8951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:46013 (error 108)
I20260810 15:20:51.132442 9541 ts_tablet_manager.cc:1939] T 4a99d6b0ae7b4b1e8e91a30293303be0 P 240a37b69cc94062a8c1f4fc92829628: Deleting consensus metadata
I20260810 15:20:51.132901 8954 catalog_manager.cc:4841] TS 240a37b69cc94062a8c1f4fc92829628 (127.4.64.67:38967): tablet 4a99d6b0ae7b4b1e8e91a30293303be0 (table test-table [id=1503d6d834284e50b81b73156eda4ed9]) successfully deleted
I20260810 15:20:51.132905 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.132749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46013 (local address 127.0.0.1:46582)
0810 15:20:51.132817 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.132876 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:46013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 15:20:51.133095 8951 catalog_manager.cc:4584] TS 1eeec2d61942436993837416b4ff75ad (127.4.64.65:46013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a99d6b0ae7b4b1e8e91a30293303be0: Network error: Client connection negotiation failed: client connection to 127.4.64.65:46013: connect: Connection refused (error 111)
I20260810 15:20:51.133172 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9187
I20260810 15:20:51.143159 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9338
I20260810 15:20:51.150905 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 8933
2026-08-10T15:20:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2238 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T15:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:51Z Disabled control of system clock
I20260810 15:20:51.186020 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:42341
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:20:51.271071 9547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:51.271309 9547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:51.271332 9547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:51.272444 9547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:51.272482 9547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:51.272495 9547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:51.272508 9547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:51.272521 9547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:51.273870 9547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:42341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:51.274084 9547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:51.274359 9547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:51.279603 9554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.279597 9552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:51.279647 9547 server_base.cc:1034] running on GCE node
W20260810 15:20:51.279654 9556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:51.280179 9547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:51.280598 9547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:51.281834 9547 hybrid_clock.cc:648] HybridClock initialized: now 1786375251281806 us; error 46 us; skew 500 ppm
I20260810 15:20:51.283322 9547 webserver.cc:460] Webserver started at http://127.4.64.126:33051/ using document root <none> and password file <none>
I20260810 15:20:51.283636 9547 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:51.283717 9547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:51.283891 9547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:51.285135 9547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "15f407c3979942d08e5c6b320e54e8fa"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "c9280bb1ed6905fdaebf9e977a545ea2"
server_key_iv: "4088e3686f2e80cfa131ec2c4ca5133f"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.285542 9547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "15f407c3979942d08e5c6b320e54e8fa"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "c9280bb1ed6905fdaebf9e977a545ea2"
server_key_iv: "4088e3686f2e80cfa131ec2c4ca5133f"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.286994 9547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:51.287775 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.287982 9547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:51.288101 9547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "15f407c3979942d08e5c6b320e54e8fa"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "c9280bb1ed6905fdaebf9e977a545ea2"
server_key_iv: "4088e3686f2e80cfa131ec2c4ca5133f"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.288183 9547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:51.292161 9547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:51.292488 9547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:51.292647 9547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:51.297160 9547 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:42341
I20260810 15:20:51.297201 9627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:42341 every 8 connection(s)
I20260810 15:20:51.297657 9547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:51.298448 9628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.300985 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9547
I20260810 15:20:51.301090 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:51.301340 4353 external_mini_cluster.cc:1428] Setting key e302219bc7432fd78495b4bd507e7488
I20260810 15:20:51.302210 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa: Bootstrap starting.
I20260810 15:20:51.302910 9628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.303270 9628 log.cc:824] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.304068 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa: No bootstrap required, opened a new log
I20260810 15:20:51.304461 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.302076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.4.64.126:42341)
0810 15:20:51.302387 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.302397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.302434 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:20:51.302905 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.302914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.303000 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.303139 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.303964 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.303968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.303969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.303974 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.304032 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.304044 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.304164 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.304211 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.304277 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260810 15:20:51.306016 9628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f407c3979942d08e5c6b320e54e8fa" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42341 } }
I20260810 15:20:51.306187 9628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.306237 9628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15f407c3979942d08e5c6b320e54e8fa, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.306373 9628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f407c3979942d08e5c6b320e54e8fa" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42341 } }
I20260810 15:20:51.306766 9628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:51.306818 9628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:51.306876 9628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:51.307452 9628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f407c3979942d08e5c6b320e54e8fa" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42341 } }
I20260810 15:20:51.307579 9628 leader_election.cc:302] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15f407c3979942d08e5c6b320e54e8fa; no voters:
I20260810 15:20:51.307771 9628 leader_election.cc:288] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:51.307896 9633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:51.308058 9633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [term 1 LEADER]: Becoming Leader. State: Replica: 15f407c3979942d08e5c6b320e54e8fa, State: Running, Role: LEADER
I20260810 15:20:51.308151 9628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:51.308313 9633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f407c3979942d08e5c6b320e54e8fa" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42341 } }
I20260810 15:20:51.308779 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15f407c3979942d08e5c6b320e54e8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f407c3979942d08e5c6b320e54e8fa" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42341 } } }
I20260810 15:20:51.308811 9635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15f407c3979942d08e5c6b320e54e8fa. Latest consensus state: current_term: 1 leader_uuid: "15f407c3979942d08e5c6b320e54e8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15f407c3979942d08e5c6b320e54e8fa" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42341 } } }
I20260810 15:20:51.308950 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:51.309011 9635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:51.309209 9643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:51.309783 9643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:51.311560 9643 catalog_manager.cc:1347] Generated new cluster ID: c1275624db954eeeb6301856a488b05f
I20260810 15:20:51.311621 9643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:51.358635 9643 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:51.358886 9643 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:51.366011 9643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa: Generated new TSK 0
I20260810 15:20:51.366269 9643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:51.381270 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:42341
--builtin_ntp_servers=127.4.64.84:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:20:51.470495 9656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:51.470760 9656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:51.470800 9656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:51.472028 9656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:51.472114 9656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:51.472167 9656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:51.473938 9656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:51.474285 9656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:51.474632 9656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:51.480432 9661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.480453 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.480432 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:51.480613 9656 server_base.cc:1034] running on GCE node
I20260810 15:20:51.480947 9656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:51.481180 9656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:51.482350 9656 hybrid_clock.cc:648] HybridClock initialized: now 1786375251482330 us; error 40 us; skew 500 ppm
I20260810 15:20:51.483811 9656 webserver.cc:460] Webserver started at http://127.4.64.65:34283/ using document root <none> and password file <none>
I20260810 15:20:51.484084 9656 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:51.484139 9656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:51.484320 9656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:51.485543 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "b99eb2a2b5e9446ab3cbea70ce71d900"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "23ec68f6ba8b6e281ace9f709844d1bc"
server_key_iv: "5069bf6c302eec8c4f056226c390d543"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.486008 9656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "b99eb2a2b5e9446ab3cbea70ce71d900"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "23ec68f6ba8b6e281ace9f709844d1bc"
server_key_iv: "5069bf6c302eec8c4f056226c390d543"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.487591 9656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:51.488512 9672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.488713 9656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:51.488848 9656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "b99eb2a2b5e9446ab3cbea70ce71d900"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "23ec68f6ba8b6e281ace9f709844d1bc"
server_key_iv: "5069bf6c302eec8c4f056226c390d543"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.488929 9656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:51.492880 9656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:51.493175 9656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:51.493331 9656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:51.493559 9656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:51.493911 9656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:51.493945 9656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.494002 9656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:51.494030 9656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.500212 9656 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:45463
I20260810 15:20:51.500299 9805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:45463 every 8 connection(s)
I20260810 15:20:51.500641 9656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:51.504019 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.501481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:53157 (local address 127.4.64.126:42341)
0810 15:20:51.501558 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.501560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.501832 (+ 272us) server_negotiation.cc:407] Connection header received
0810 15:20:51.502359 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.502360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.502366 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.502432 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.503718 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.503719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.503719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.503723 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.503739 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.503744 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.503803 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.503946 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.503961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:20:51.504148 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.501472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:42341 (local address 127.4.64.65:53157)
0810 15:20:51.501696 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.501740 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.502264 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.502485 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.502489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.502498 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:51.503557 (+ 1059us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:51.503557 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:51.503586 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:51.503614 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:51.503811 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:51.503816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:51.503819 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.503909 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.503962 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260810 15:20:51.504732 9806 heartbeater.cc:347] Connected to a master server at 127.4.64.126:42341
I20260810 15:20:51.504815 9806 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:51.504990 9806 heartbeater.cc:511] Master 127.4.64.126:42341 requested a full tablet report, sending...
I20260810 15:20:51.505679 9581 ts_manager.cc:194] Registered new tserver with Master: b99eb2a2b5e9446ab3cbea70ce71d900 (127.4.64.65:45463)
I20260810 15:20:51.508001 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9656
I20260810 15:20:51.508121 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:51.508353 4353 external_mini_cluster.cc:1428] Setting key 09c642dc90a1440230e4b55ab26efb96
I20260810 15:20:51.509364 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:42341
--builtin_ntp_servers=127.4.64.84:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:20:51.595059 9809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:51.595283 9809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:51.595307 9809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:51.596550 9809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:51.596596 9809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:51.596623 9809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:51.598100 9809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:51.598379 9809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:51.598711 9809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:51.604151 9815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.604151 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.604374 9816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:51.604460 9809 server_base.cc:1034] running on GCE node
I20260810 15:20:51.604832 9809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:51.605151 9809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:51.606320 9809 hybrid_clock.cc:648] HybridClock initialized: now 1786375251606304 us; error 29 us; skew 500 ppm
I20260810 15:20:51.607754 9809 webserver.cc:460] Webserver started at http://127.4.64.66:34367/ using document root <none> and password file <none>
I20260810 15:20:51.608016 9809 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:51.608064 9809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:51.608321 9809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:51.609587 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "03041e12576e44859f94d0d1a9e6a22a"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "1338da68e2645408e57d99142ce5d0a8"
server_key_iv: "0ad1918f7d00b0f04e43425f489e9c5e"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.610069 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "03041e12576e44859f94d0d1a9e6a22a"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "1338da68e2645408e57d99142ce5d0a8"
server_key_iv: "0ad1918f7d00b0f04e43425f489e9c5e"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.611492 9809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:51.612378 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.612581 9809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:51.612681 9809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "03041e12576e44859f94d0d1a9e6a22a"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "1338da68e2645408e57d99142ce5d0a8"
server_key_iv: "0ad1918f7d00b0f04e43425f489e9c5e"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.612767 9809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:51.616420 9809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:51.616726 9809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:51.616883 9809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:51.617147 9809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:51.617487 9809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:51.617522 9809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.617573 9809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:51.617599 9809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.624821 9809 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:35979
I20260810 15:20:51.624909 9956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:35979 every 8 connection(s)
I20260810 15:20:51.625303 9809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:51.628914 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.626199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:40203 (local address 127.4.64.126:42341)
0810 15:20:51.626352 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.626354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.626620 (+ 266us) server_negotiation.cc:407] Connection header received
0810 15:20:51.627190 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.627191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.627197 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.627280 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.628604 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.628605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.628605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.628610 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.628625 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.628630 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.628721 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.628838 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.628854 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260810 15:20:51.629109 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.626287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:42341 (local address 127.4.64.66:40203)
0810 15:20:51.626471 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.626514 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.627000 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.627356 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.627359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.627369 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:51.628414 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:51.628414 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:51.628445 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:51.628482 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:51.628734 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:51.628738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:51.628741 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.628805 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.628859 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260810 15:20:51.629799 9957 heartbeater.cc:347] Connected to a master server at 127.4.64.126:42341
I20260810 15:20:51.629900 9957 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:51.630021 9957 heartbeater.cc:511] Master 127.4.64.126:42341 requested a full tablet report, sending...
I20260810 15:20:51.630426 9581 ts_manager.cc:194] Registered new tserver with Master: 03041e12576e44859f94d0d1a9e6a22a (127.4.64.66:35979)
I20260810 15:20:51.635386 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9809
I20260810 15:20:51.635488 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:51.635736 4353 external_mini_cluster.cc:1428] Setting key 3912f042c84e7e22cf57b33e06cffa82
I20260810 15:20:51.637108 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:42341
--builtin_ntp_servers=127.4.64.84:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:20:51.723524 9961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:51.723841 9961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:51.723893 9961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:51.725255 9961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:51.725306 9961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:51.725334 9961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:51.726809 9961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:51.727108 9961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:51.727466 9961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:51.732888 9975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.732992 9969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:51.732884 9968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:51.733115 9961 server_base.cc:1034] running on GCE node
I20260810 15:20:51.733314 9961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:51.733592 9961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:51.734777 9961 hybrid_clock.cc:648] HybridClock initialized: now 1786375251734767 us; error 32 us; skew 500 ppm
I20260810 15:20:51.736099 9961 webserver.cc:460] Webserver started at http://127.4.64.67:44097/ using document root <none> and password file <none>
I20260810 15:20:51.736366 9961 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:51.736438 9961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:51.736600 9961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:51.737794 9961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "270cfb85035647a3acf90f93d657b7ed"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "50fb1713024a608542dd5f1798379d7a"
server_key_iv: "042318fcc095c2101c81ea514ff0c099"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.738193 9961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "270cfb85035647a3acf90f93d657b7ed"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "50fb1713024a608542dd5f1798379d7a"
server_key_iv: "042318fcc095c2101c81ea514ff0c099"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.739688 9961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:20:51.740532 9982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.740720 9961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:51.740844 9961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "270cfb85035647a3acf90f93d657b7ed"
format_stamp: "Formatted at 2026-08-10 15:20:51 on dist-test-slave-9zdj"
server_key: "50fb1713024a608542dd5f1798379d7a"
server_key_iv: "042318fcc095c2101c81ea514ff0c099"
server_key_version: "encryptionkey@0"
I20260810 15:20:51.740923 9961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:51.744464 9961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:51.744750 9961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:51.744910 9961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:51.745146 9961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:51.745478 9961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:51.745534 9961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.745584 9961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:51.745662 9961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:51.751683 9961 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:34893
I20260810 15:20:51.751729 10109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:34893 every 8 connection(s)
I20260810 15:20:51.752090 9961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:51.753376 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 9961
I20260810 15:20:51.753463 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:51.753723 4353 external_mini_cluster.cc:1428] Setting key 7ad13d3928604aaf68f7753db21db750
I20260810 15:20:51.755738 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.752923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:33209 (local address 127.4.64.126:42341)
0810 15:20:51.753000 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.753002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.753251 (+ 249us) server_negotiation.cc:407] Connection header received
0810 15:20:51.753966 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.753967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.753974 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.754068 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.755529 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.755531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.755531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.755535 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.755549 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.755553 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.755600 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.755672 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.755684 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:20:51.755991 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.752862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:42341 (local address 127.4.64.67:33209)
0810 15:20:51.753078 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.753124 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.753833 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.754143 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.754148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.754159 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:51.755372 (+ 1213us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:51.755372 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:51.755406 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:51.755437 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:51.755602 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:51.755604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:51.755607 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.755672 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.755726 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260810 15:20:51.756498 10110 heartbeater.cc:347] Connected to a master server at 127.4.64.126:42341
I20260810 15:20:51.756587 10110 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:51.756758 10110 heartbeater.cc:511] Master 127.4.64.126:42341 requested a full tablet report, sending...
I20260810 15:20:51.757104 9581 ts_manager.cc:194] Registered new tserver with Master: 270cfb85035647a3acf90f93d657b7ed (127.4.64.67:34893)
I20260810 15:20:51.766184 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:20:51.768921 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.767723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.4.64.126:42341)
0810 15:20:51.767803 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.767805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.767913 (+ 108us) server_negotiation.cc:407] Connection header received
0810 15:20:51.768013 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.768014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.768020 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.768081 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.768683 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.768685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.768685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.768689 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.768705 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.768709 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.768766 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.768821 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.768836 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:20:51.770783 9581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:20:51.772603 9581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:51.777215 10126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.775143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45463 (local address 127.0.0.1:49192)
0810 15:20:51.775521 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.775533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.775591 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.776099 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.776100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.776103 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:51.776733 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:51.776734 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:51.776742 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:51.776770 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:51.777018 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:51.777025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:51.777029 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.777101 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.777116 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":114,"threads_started":1}
I20260810 15:20:51.777361 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.775208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.4.64.67:34893)
0810 15:20:51.775552 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.775555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.775565 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:51.775594 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.775596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.775606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.775684 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.776983 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.776992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.776992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.776997 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.777041 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.777068 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.777126 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.777248 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.777264 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":86,"threads_started":1}
I20260810 15:20:51.777451 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.775666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.4.64.66:35979)
0810 15:20:51.775882 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.775887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.776038 (+ 151us) server_negotiation.cc:407] Connection header received
0810 15:20:51.776072 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.776074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.776084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.776167 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.777065 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.777070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.777070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.777095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.777129 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.777146 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.777205 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.777354 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.777372 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
I20260810 15:20:51.777475 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.775116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:34893 (local address 127.0.0.1:42994)
0810 15:20:51.775348 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.775398 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.775461 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.776059 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.776066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.776073 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:51.776887 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:51.776887 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:51.776892 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:51.776899 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:51.777151 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:51.777152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:51.777153 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.777195 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.777206 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260810 15:20:51.777789 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.775374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:35979 (local address 127.0.0.1:35630)
0810 15:20:51.775937 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:51.775948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:51.776002 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:51.776179 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:51.776180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:51.776181 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:51.776720 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:51.776721 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:51.776763 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:51.776771 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:51.777239 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:51.777240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:51.777241 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:20:51.777705 (+ 464us) client_negotiation.cc:240] Negotiation successful
0810 15:20:51.777724 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":94,"threads_started":1}
I20260810 15:20:51.777256 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:51.775272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.4.64.65:45463)
0810 15:20:51.775457 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:51.775463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:51.775660 (+ 197us) server_negotiation.cc:407] Connection header received
0810 15:20:51.775693 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:51.775696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:51.775707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:51.775780 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:51.776862 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:51.776874 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:51.776874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:51.776880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:51.776923 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:51.776954 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:51.777028 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:51.777131 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 15:20:51.777149 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260810 15:20:51.778257 10037 tablet_service.cc:1467] Processing CreateTablet for tablet 8986c978454a480dabbd08149b772dfb (DEFAULT_TABLE table=test-table [id=efc79b53430e4e2cbe287fd03d69bfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:51.778409 9875 tablet_service.cc:1467] Processing CreateTablet for tablet 8986c978454a480dabbd08149b772dfb (DEFAULT_TABLE table=test-table [id=efc79b53430e4e2cbe287fd03d69bfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:51.778668 10037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986c978454a480dabbd08149b772dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.778712 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986c978454a480dabbd08149b772dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.778566 9725 tablet_service.cc:1467] Processing CreateTablet for tablet 8986c978454a480dabbd08149b772dfb (DEFAULT_TABLE table=test-table [id=efc79b53430e4e2cbe287fd03d69bfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:51.778947 9725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8986c978454a480dabbd08149b772dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:51.781585 10128 tablet_bootstrap.cc:492] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: Bootstrap starting.
I20260810 15:20:51.782174 10130 tablet_bootstrap.cc:492] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Bootstrap starting.
I20260810 15:20:51.782579 10128 tablet_bootstrap.cc:654] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.782944 10128 log.cc:824] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.782972 10130 tablet_bootstrap.cc:654] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.783306 10130 log.cc:824] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.783398 10129 tablet_bootstrap.cc:492] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Bootstrap starting.
I20260810 15:20:51.783871 10128 tablet_bootstrap.cc:492] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: No bootstrap required, opened a new log
I20260810 15:20:51.783965 10128 ts_tablet_manager.cc:1397] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:20:51.784049 10129 tablet_bootstrap.cc:654] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:51.784126 10130 tablet_bootstrap.cc:492] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: No bootstrap required, opened a new log
I20260810 15:20:51.784197 10130 ts_tablet_manager.cc:1397] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:51.784416 10129 log.cc:824] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:51.785285 10129 tablet_bootstrap.cc:492] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: No bootstrap required, opened a new log
I20260810 15:20:51.785356 10129 ts_tablet_manager.cc:1397] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:20:51.786548 10128 raft_consensus.cc:348] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:51.786710 10128 raft_consensus.cc:374] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.786754 10128 raft_consensus.cc:729] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b99eb2a2b5e9446ab3cbea70ce71d900, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.786875 10128 consensus_queue.cc:260] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:51.786959 10130 raft_consensus.cc:348] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:51.787128 10130 raft_consensus.cc:374] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.787168 10128 ts_tablet_manager.cc:1428] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260810 15:20:51.787175 10130 raft_consensus.cc:729] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03041e12576e44859f94d0d1a9e6a22a, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.787302 9806 heartbeater.cc:503] Master 127.4.64.126:42341 was elected leader, sending a full tablet report...
I20260810 15:20:51.787295 10130 consensus_queue.cc:260] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:51.787596 10130 ts_tablet_manager.cc:1428] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:20:51.787613 9957 heartbeater.cc:503] Master 127.4.64.126:42341 was elected leader, sending a full tablet report...
I20260810 15:20:51.788209 10129 raft_consensus.cc:348] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:51.788383 10129 raft_consensus.cc:374] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:51.788439 10129 raft_consensus.cc:729] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270cfb85035647a3acf90f93d657b7ed, State: Initialized, Role: FOLLOWER
I20260810 15:20:51.788568 10129 consensus_queue.cc:260] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:51.788854 10129 ts_tablet_manager.cc:1428] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260810 15:20:51.788980 10110 heartbeater.cc:503] Master 127.4.64.126:42341 was elected leader, sending a full tablet report...
W20260810 15:20:51.876010 9958 tablet.cc:2369] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:20:52.001415 9807 tablet.cc:2369] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:20:52.002687 10111 tablet.cc:2369] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:52.060715 10134 raft_consensus.cc:482] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:52.060861 10134 raft_consensus.cc:504] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:52.061353 10134 leader_election.cc:288] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 270cfb85035647a3acf90f93d657b7ed (127.4.64.67:34893), 03041e12576e44859f94d0d1a9e6a22a (127.4.64.66:35979)
I20260810 15:20:52.062873 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.061549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:34893 (local address 127.4.64.65:42619)
0810 15:20:52.061713 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:52.061725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:52.061802 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:52.062027 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:52.062028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:52.062031 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:52.062481 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:52.062482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:52.062489 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:52.062500 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:52.062683 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:52.062684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:52.062684 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 15:20:52.062736 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:20:52.062751 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":117,"threads_started":1}
I20260810 15:20:52.062966 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.061655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:42619 (local address 127.4.64.67:34893)
0810 15:20:52.061734 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.061736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.061895 (+ 159us) server_negotiation.cc:407] Connection header received
0810 15:20:52.061926 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.061927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.061940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.062017 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:52.062595 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:52.062596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:52.062596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:52.062600 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:52.062616 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:52.062621 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:52.062686 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:52.062873 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 15:20:52.062888 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:20:52.063031 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.061803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:35979 (local address 127.4.64.65:50415)
0810 15:20:52.061896 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:52.061905 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:52.061935 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:52.062125 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:52.062126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:52.062128 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:52.062648 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:20:52.062648 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:52.062656 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:20:52.062676 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:52.062851 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:52.062852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:52.062853 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:20:52.062933 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:20:52.062949 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":42,"threads_started":1}
I20260810 15:20:52.063026 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.061960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:50415 (local address 127.4.64.66:35979)
0810 15:20:52.062014 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.062016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.062023 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:52.062055 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.062056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.062060 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.062099 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:52.062743 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:52.062745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:52.062745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:52.062749 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:52.062764 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:52.062769 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:52.062842 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:52.062946 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 15:20:52.062961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:20:52.063277 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986c978454a480dabbd08149b772dfb" candidate_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270cfb85035647a3acf90f93d657b7ed" is_pre_election: true
I20260810 15:20:52.063407 9907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986c978454a480dabbd08149b772dfb" candidate_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03041e12576e44859f94d0d1a9e6a22a" is_pre_election: true
I20260810 15:20:52.063454 10059 raft_consensus.cc:2454] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b99eb2a2b5e9446ab3cbea70ce71d900 in term 0.
I20260810 15:20:52.063545 9907 raft_consensus.cc:2454] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b99eb2a2b5e9446ab3cbea70ce71d900 in term 0.
I20260810 15:20:52.063716 9677 leader_election.cc:302] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 270cfb85035647a3acf90f93d657b7ed, b99eb2a2b5e9446ab3cbea70ce71d900; no voters:
I20260810 15:20:52.063882 10134 raft_consensus.cc:2781] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:52.063963 10134 raft_consensus.cc:482] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:52.063982 10134 raft_consensus.cc:3037] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:52.064752 10134 raft_consensus.cc:504] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:52.064898 10134 leader_election.cc:288] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [CANDIDATE]: Term 1 election: Requested vote from peers 270cfb85035647a3acf90f93d657b7ed (127.4.64.67:34893), 03041e12576e44859f94d0d1a9e6a22a (127.4.64.66:35979)
I20260810 15:20:52.065197 9907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986c978454a480dabbd08149b772dfb" candidate_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03041e12576e44859f94d0d1a9e6a22a"
I20260810 15:20:52.065254 10059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8986c978454a480dabbd08149b772dfb" candidate_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270cfb85035647a3acf90f93d657b7ed"
I20260810 15:20:52.065302 9907 raft_consensus.cc:3037] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:52.065330 10059 raft_consensus.cc:3037] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:52.066327 10059 raft_consensus.cc:2454] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b99eb2a2b5e9446ab3cbea70ce71d900 in term 1.
I20260810 15:20:52.066336 9907 raft_consensus.cc:2454] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b99eb2a2b5e9446ab3cbea70ce71d900 in term 1.
I20260810 15:20:52.066500 9677 leader_election.cc:302] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03041e12576e44859f94d0d1a9e6a22a, b99eb2a2b5e9446ab3cbea70ce71d900; no voters:
I20260810 15:20:52.066609 10134 raft_consensus.cc:2781] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:52.066776 10134 raft_consensus.cc:686] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [term 1 LEADER]: Becoming Leader. State: Replica: b99eb2a2b5e9446ab3cbea70ce71d900, State: Running, Role: LEADER
I20260810 15:20:52.066883 10134 consensus_queue.cc:237] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } }
I20260810 15:20:52.067826 9580 catalog_manager.cc:5495] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 reported cstate change: term changed from 0 to 1, leader changed from <none> to b99eb2a2b5e9446ab3cbea70ce71d900 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b99eb2a2b5e9446ab3cbea70ce71d900" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45463 } health_report { overall_health: HEALTHY } } }
I20260810 15:20:52.134970 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.133902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.4.64.65:45463)
0810 15:20:52.133967 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.133969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.133979 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:52.134021 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.134022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.134046 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.134106 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:52.134664 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:52.134665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:52.134665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:52.134669 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:52.134685 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:52.134690 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:52.134745 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:52.134806 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 15:20:52.134842 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 15:20:52.137086 9907 raft_consensus.cc:1261] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 1 FOLLOWER]: Refusing update from remote peer b99eb2a2b5e9446ab3cbea70ce71d900: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:52.137106 10059 raft_consensus.cc:1261] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 1 FOLLOWER]: Refusing update from remote peer b99eb2a2b5e9446ab3cbea70ce71d900: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:52.137494 10134 consensus_queue.cc:1035] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "270cfb85035647a3acf90f93d657b7ed" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 34893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:52.137728 10139 consensus_queue.cc:1035] T 8986c978454a480dabbd08149b772dfb P b99eb2a2b5e9446ab3cbea70ce71d900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03041e12576e44859f94d0d1a9e6a22a" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:52.140395 10143 mvcc.cc:205] Tried to move back new op lower bound from 7316993032750280704 to 7316993032466485248. Current Snapshot: MvccSnapshot[applied={T|T < 7316993032750280704}]
I20260810 15:20:52.142503 9580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:52.142630 9580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:52.143594 9580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15f407c3979942d08e5c6b320e54e8fa: Sending DeleteTablet for 3 replicas of tablet 8986c978454a480dabbd08149b772dfb
I20260810 15:20:52.143985 10037 tablet_service.cc:1514] Processing DeleteTablet for tablet 8986c978454a480dabbd08149b772dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:52 UTC) from {username='slave'} at 127.0.0.1:42994
I20260810 15:20:52.143978 9875 tablet_service.cc:1514] Processing DeleteTablet for tablet 8986c978454a480dabbd08149b772dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:52 UTC) from {username='slave'} at 127.0.0.1:35630
I20260810 15:20:52.144349 9725 tablet_service.cc:1514] Processing DeleteTablet for tablet 8986c978454a480dabbd08149b772dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:52 UTC) from {username='slave'} at 127.0.0.1:49192
I20260810 15:20:52.144244 10155 tablet_replica.cc:330] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: stopping tablet replica
I20260810 15:20:52.144662 10155 raft_consensus.cc:2229] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:52.144658 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9656
I20260810 15:20:52.146142 10155 raft_consensus.cc:2258] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:52.146339 10157 tablet_replica.cc:330] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: stopping tablet replica
I20260810 15:20:52.146446 10157 raft_consensus.cc:2229] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:52.146564 10157 raft_consensus.cc:2258] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:52.149266 10155 ts_tablet_manager.cc:1905] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:52.151559 10155 ts_tablet_manager.cc:1918] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:52.151635 10155 log.cc:1196] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/8986c978454a480dabbd08149b772dfb
I20260810 15:20:52.151973 10155 ts_tablet_manager.cc:1939] T 8986c978454a480dabbd08149b772dfb P 270cfb85035647a3acf90f93d657b7ed: Deleting consensus metadata
I20260810 15:20:52.152184 10157 ts_tablet_manager.cc:1905] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:52.153667 10157 ts_tablet_manager.cc:1918] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:52.153753 10157 log.cc:1196] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/8986c978454a480dabbd08149b772dfb
I20260810 15:20:52.154021 10157 ts_tablet_manager.cc:1939] T 8986c978454a480dabbd08149b772dfb P 03041e12576e44859f94d0d1a9e6a22a: Deleting consensus metadata
I20260810 15:20:52.154512 9564 catalog_manager.cc:4841] TS 03041e12576e44859f94d0d1a9e6a22a (127.4.64.66:35979): tablet 8986c978454a480dabbd08149b772dfb (table test-table [id=efc79b53430e4e2cbe287fd03d69bfdd]) successfully deleted
W20260810 15:20:52.154685 9563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:45463 (error 108)
I20260810 15:20:52.154882 9564 catalog_manager.cc:4841] TS 270cfb85035647a3acf90f93d657b7ed (127.4.64.67:34893): tablet 8986c978454a480dabbd08149b772dfb (table test-table [id=efc79b53430e4e2cbe287fd03d69bfdd]) successfully deleted
I20260810 15:20:52.155032 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9809
I20260810 15:20:52.156555 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.155112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45463 (local address 127.0.0.1:49210)
0810 15:20:52.156474 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:52.156523 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1340}
W20260810 15:20:52.156931 9563 catalog_manager.cc:4584] TS b99eb2a2b5e9446ab3cbea70ce71d900 (127.4.64.65:45463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8986c978454a480dabbd08149b772dfb: Network error: Client connection negotiation failed: client connection to 127.4.64.65:45463: connect: Connection refused (error 111)
I20260810 15:20:52.163448 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9961
I20260810 15:20:52.172976 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 9547
2026-08-10T15:20:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1020 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 15:20:52.197562 4353 mini_oidc.cc:152] Starting JWKS server
I20260810 15:20:52.198828 4353 webserver.cc:460] Webserver started at https://127.0.0.1:37813/ using document root <none> and password file <none>
I20260810 15:20:52.198968 4353 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 15:20:52.199168 4353 webserver.cc:460] Webserver started at http://127.0.0.1:41305/ using document root <none> and password file <none>
2026-08-10T15:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:52Z Disabled control of system clock
I20260810 15:20:52.209470 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:42941
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:34801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:42941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37813/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/testchainca.cert with env {}
W20260810 15:20:52.295291 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:52.295519 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:52.295544 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:52.296702 10165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:52.296741 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:52.296752 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:52.296765 10165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:52.296777 10165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:20:52.296789 10165 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:20:52.296801 10165 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37813/jwks.json
I20260810 15:20:52.298341 10165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:42941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:42941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37813/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:52.298563 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:52.298769 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:52.304251 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:52.304280 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:52.304280 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:52.304481 10165 server_base.cc:1034] running on GCE node
I20260810 15:20:52.304855 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:52.305208 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:52.306370 10165 hybrid_clock.cc:648] HybridClock initialized: now 1786375252306345 us; error 38 us; skew 500 ppm
I20260810 15:20:52.307675 10165 webserver.cc:460] Webserver started at http://127.4.64.126:37625/ using document root <none> and password file <none>
I20260810 15:20:52.307947 10165 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:52.307997 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:52.308183 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:52.309321 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "42f5625bf2634160bf9d0c8c2cdeddc4"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "00f85c8a2eecedf67785dbec53a72416"
server_key_iv: "83748746fcbf5f25fe5c2a1d191f8ded"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.309753 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "42f5625bf2634160bf9d0c8c2cdeddc4"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "00f85c8a2eecedf67785dbec53a72416"
server_key_iv: "83748746fcbf5f25fe5c2a1d191f8ded"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.311424 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:52.312361 10179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:52.312582 10165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:52.312683 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "42f5625bf2634160bf9d0c8c2cdeddc4"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "00f85c8a2eecedf67785dbec53a72416"
server_key_iv: "83748746fcbf5f25fe5c2a1d191f8ded"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.312736 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:52.321162 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:52.321456 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:52.321563 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:52.325817 10165 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:42941
I20260810 15:20:52.325855 10238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:42941 every 8 connection(s)
I20260810 15:20:52.326234 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:52.327381 10239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:52.329918 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4: Bootstrap starting.
I20260810 15:20:52.330963 10239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:52.331239 10239 log.cc:824] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:52.332005 10239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4: No bootstrap required, opened a new log
I20260810 15:20:52.333777 10239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42941 } }
I20260810 15:20:52.333923 10239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:52.333946 10239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f5625bf2634160bf9d0c8c2cdeddc4, State: Initialized, Role: FOLLOWER
I20260810 15:20:52.334055 10239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42941 } }
I20260810 15:20:52.334133 10239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:52.334188 10239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:52.334261 10239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:52.334801 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10165
I20260810 15:20:52.334872 10239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42941 } }
I20260810 15:20:52.334894 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:52.334982 10239 leader_election.cc:302] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f5625bf2634160bf9d0c8c2cdeddc4; no voters:
I20260810 15:20:52.335129 4353 external_mini_cluster.cc:1428] Setting key 2ad276a004c6c7dc5daff1c6798d0e3c
I20260810 15:20:52.335208 10239 leader_election.cc:288] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:52.335377 10243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:52.335525 10239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:52.335598 10243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [term 1 LEADER]: Becoming Leader. State: Replica: 42f5625bf2634160bf9d0c8c2cdeddc4, State: Running, Role: LEADER
I20260810 15:20:52.335788 10243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42941 } }
I20260810 15:20:52.336371 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f5625bf2634160bf9d0c8c2cdeddc4. Latest consensus state: current_term: 1 leader_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42941 } } }
I20260810 15:20:52.336473 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:52.336833 10244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f5625bf2634160bf9d0c8c2cdeddc4" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42941 } } }
I20260810 15:20:52.336911 10244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:52.337663 10264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:52.337720 10264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:52.338439 10255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:52.338660 10255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:52.340502 10255 catalog_manager.cc:1347] Generated new cluster ID: d941b25a35314dcdb6e7a3cf73ea3fc3
I20260810 15:20:52.340560 10255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:52.341866 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.335696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.4.64.126:42941)
0810 15:20:52.336076 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.336086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.336123 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:20:52.336250 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.336258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.336332 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.336763 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:52.336887 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.338292 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.339423 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.340103 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.340251 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.340992 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:52.340996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:52.340996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:52.341218 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:52.341262 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:52.341276 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:52.341511 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:52.341542 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 15:20:52.341649 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":71,"threads_started":1}
I20260810 15:20:52.343236 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:52.351548 10255 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:52.352149 10255 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:52.359056 10255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f5625bf2634160bf9d0c8c2cdeddc4: Generated new TSK 0
I20260810 15:20:52.359297 10255 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 15:20:52.391924 10274 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 15:20:52.392086 10274 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 15:20:52.392297 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.368605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.4.64.126:42941)
0810 15:20:52.368681 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.368684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.368845 (+ 161us) server_negotiation.cc:407] Connection header received
0810 15:20:52.368982 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.368983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.368995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.369078 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:20:52.369253 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.369826 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.370651 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.370850 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.370939 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.371179 (+ 240us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:20:52.391991 (+ 20812us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 15:20:52.392224 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 15:20:52.392469 10248 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 15:20:52.392722 10273 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:52.368533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:42941 (local address 127.0.0.1:34458)
0810 15:20:52.368767 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:52.368777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:52.368865 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:52.369090 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:52.369091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:52.369093 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:20:52.369177 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:52.369180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.369914 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.369915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:52.370555 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:52.370557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.370943 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.370944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:52.371013 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.371043 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:20:52.392225 (+ 21182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 15:20:52.392376 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.64.126:42941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.64.126:42941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 15:20:52.393214 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10165
2026-08-10T15:20:52Z chronyd exiting
I20260810 15:20:52.433388 4353 test_util.cc:175] -----------------------------------------------
I20260810 15:20:52.433480 4353 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786375228510355-4353-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 15:20:52.434389 4353 mini_oidc.cc:152] Starting JWKS server
I20260810 15:20:52.435142 4353 webserver.cc:460] Webserver started at https://127.0.0.1:43089/ using document root <none> and password file <none>
I20260810 15:20:52.435185 4353 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 15:20:52.435356 4353 webserver.cc:460] Webserver started at http://127.0.0.1:41973/ using document root <none> and password file <none>
2026-08-10T15:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:52Z Disabled control of system clock
I20260810 15:20:52.445693 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:33487
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:33487
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43089/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/testchainca.cert with env {}
W20260810 15:20:52.531514 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:52.531749 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:52.531772 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:52.533004 10280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:52.533053 10280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:52.533066 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:52.533079 10280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:52.533092 10280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:20:52.533104 10280 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:20:52.533161 10280 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43089/jwks.json
I20260810 15:20:52.534777 10280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:33487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:33487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43089/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:52.535063 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:52.535295 10280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:52.540776 10286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:52.540830 10288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:52.540858 10290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:52.541085 10280 server_base.cc:1034] running on GCE node
I20260810 15:20:52.541376 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:52.541750 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:52.542900 10280 hybrid_clock.cc:648] HybridClock initialized: now 1786375252542878 us; error 38 us; skew 500 ppm
I20260810 15:20:52.544234 10280 webserver.cc:460] Webserver started at http://127.4.64.126:37537/ using document root <none> and password file <none>
I20260810 15:20:52.544507 10280 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:52.544557 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:52.544711 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:52.545907 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "6e4e404242094591969c3bb9f8ccdaf5"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "032c7c229469ff939357179b37db1531"
server_key_iv: "0011193acd613bd746e8dd90b3ea6424"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.546296 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "6e4e404242094591969c3bb9f8ccdaf5"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "032c7c229469ff939357179b37db1531"
server_key_iv: "0011193acd613bd746e8dd90b3ea6424"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.547888 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:52.548756 10298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:52.548997 10280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:52.549111 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "6e4e404242094591969c3bb9f8ccdaf5"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "032c7c229469ff939357179b37db1531"
server_key_iv: "0011193acd613bd746e8dd90b3ea6424"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.549192 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:52.577131 10280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:52.577519 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:52.577732 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:52.582876 10280 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:33487
I20260810 15:20:52.583040 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:33487 every 8 connection(s)
I20260810 15:20:52.583362 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:52.584611 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:52.587436 10356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5: Bootstrap starting.
I20260810 15:20:52.588063 10356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:52.588351 10356 log.cc:824] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:52.589161 10356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5: No bootstrap required, opened a new log
I20260810 15:20:52.591248 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10280
I20260810 15:20:52.591370 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:52.591349 10356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4e404242094591969c3bb9f8ccdaf5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33487 } }
I20260810 15:20:52.591572 10356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:52.591598 4353 external_mini_cluster.cc:1428] Setting key 29065608be43d5b9b97d3db11df13f1b
I20260810 15:20:52.591615 10356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4e404242094591969c3bb9f8ccdaf5, State: Initialized, Role: FOLLOWER
I20260810 15:20:52.591842 10356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4e404242094591969c3bb9f8ccdaf5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33487 } }
I20260810 15:20:52.591956 10356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:52.592042 10356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:52.592116 10356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:52.592906 10356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4e404242094591969c3bb9f8ccdaf5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33487 } }
I20260810 15:20:52.593051 10356 leader_election.cc:302] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4e404242094591969c3bb9f8ccdaf5; no voters:
I20260810 15:20:52.593251 10356 leader_election.cc:288] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:52.593272 10361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:52.593506 10361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [term 1 LEADER]: Becoming Leader. State: Replica: 6e4e404242094591969c3bb9f8ccdaf5, State: Running, Role: LEADER
I20260810 15:20:52.593559 10356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:52.593684 10361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4e404242094591969c3bb9f8ccdaf5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33487 } }
I20260810 15:20:52.594125 10362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e4e404242094591969c3bb9f8ccdaf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4e404242094591969c3bb9f8ccdaf5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33487 } } }
I20260810 15:20:52.594259 10362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:52.594455 10363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e4e404242094591969c3bb9f8ccdaf5. Latest consensus state: current_term: 1 leader_uuid: "6e4e404242094591969c3bb9f8ccdaf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4e404242094591969c3bb9f8ccdaf5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33487 } } }
I20260810 15:20:52.594576 10363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:52.594971 10370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:52.595517 10370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:52.597373 10370 catalog_manager.cc:1347] Generated new cluster ID: 88d09c87f7574d7384305c61069138f0
I20260810 15:20:52.597465 10370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:52.598847 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.592347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.4.64.126:33487)
0810 15:20:52.592612 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.592621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.592648 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:20:52.592754 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.592760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.592829 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.592976 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:52.593219 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.595110 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.596377 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.596987 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.597118 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.598054 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:52.598058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:52.598059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:52.598270 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:52.598307 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:52.598319 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:52.598416 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:52.598506 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:20:52.598597 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":116,"threads_started":1}
I20260810 15:20:52.599983 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:52.612962 10370 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:52.613785 10370 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:52.621531 10370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e4e404242094591969c3bb9f8ccdaf5: Generated new TSK 0
I20260810 15:20:52.621784 10370 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 15:20:52.652510 10389 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 15:20:52.652674 10389 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 15:20:52.653064 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.629361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.4.64.126:33487)
0810 15:20:52.629407 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.629409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.629485 (+ 76us) server_negotiation.cc:407] Connection header received
0810 15:20:52.629578 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.629579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.629589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.629702 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:20:52.629834 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.630333 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.631509 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.631725 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.631785 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.631980 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:20:52.652759 (+ 20779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 15:20:52.652930 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50764: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:20:52.653146 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:52.629235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:33487 (local address 127.0.0.1:50764)
0810 15:20:52.629414 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:52.629428 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:52.629499 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:52.629700 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:52.629700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:52.629702 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:20:52.629766 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:52.629769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.630423 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.630424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:52.631409 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:52.631413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.631803 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.631804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:52.631871 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.631876 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:20:52.652886 (+ 21010us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 15:20:52.653015 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.64.126:33487: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
W20260810 15:20:52.653172 10360 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50764: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43089/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260810 15:20:52.653532 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10280
2026-08-10T15:20:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (261 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 15:20:52.696367 4353 mini_oidc.cc:152] Starting JWKS server
I20260810 15:20:52.697213 4353 webserver.cc:460] Webserver started at https://127.0.0.1:34799/ using document root <none> and password file <none>
I20260810 15:20:52.697300 4353 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 15:20:52.697530 4353 webserver.cc:460] Webserver started at http://127.0.0.1:37651/ using document root <none> and password file <none>
2026-08-10T15:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:52Z Disabled control of system clock
I20260810 15:20:52.707499 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35107
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:35107
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34799/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 15:20:52.793174 10398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:52.793424 10398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:52.793460 10398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:52.794708 10398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:52.794782 10398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:52.794804 10398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:52.794824 10398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:52.794852 10398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:20:52.794884 10398 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:20:52.794919 10398 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34799/jwks.json
I20260810 15:20:52.796417 10398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:35107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34799/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:52.796721 10398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:52.797047 10398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:52.802744 10404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:52.802744 10403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:52.802776 10406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:52.803223 10398 server_base.cc:1034] running on GCE node
I20260810 15:20:52.803558 10398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:52.803876 10398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:52.805045 10398 hybrid_clock.cc:648] HybridClock initialized: now 1786375252805031 us; error 48 us; skew 500 ppm
I20260810 15:20:52.806546 10398 webserver.cc:460] Webserver started at http://127.4.64.126:33017/ using document root <none> and password file <none>
I20260810 15:20:52.806839 10398 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:52.806914 10398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:52.807073 10398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:52.808214 10398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "dfbf51c241454ec18ccdb48945c09406"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "2e2c8abc57f10a9c98aa97e870845b8b"
server_key_iv: "8104784accfe311d74050f0d227b9eff"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.808590 10398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "dfbf51c241454ec18ccdb48945c09406"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "2e2c8abc57f10a9c98aa97e870845b8b"
server_key_iv: "8104784accfe311d74050f0d227b9eff"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.810035 10398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:52.810945 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:52.811290 10398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:52.811416 10398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "dfbf51c241454ec18ccdb48945c09406"
format_stamp: "Formatted at 2026-08-10 15:20:52 on dist-test-slave-9zdj"
server_key: "2e2c8abc57f10a9c98aa97e870845b8b"
server_key_iv: "8104784accfe311d74050f0d227b9eff"
server_key_version: "encryptionkey@0"
I20260810 15:20:52.811502 10398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:52.827911 10398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:52.828337 10398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:52.828537 10398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:52.833519 10398 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:35107
I20260810 15:20:52.833575 10478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:35107 every 8 connection(s)
I20260810 15:20:52.834033 10398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:52.834827 10479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:52.837868 10479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406: Bootstrap starting.
I20260810 15:20:52.838449 10479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:52.838713 10479 log.cc:824] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:52.839512 10479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406: No bootstrap required, opened a new log
I20260810 15:20:52.841193 10479 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf51c241454ec18ccdb48945c09406" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35107 } }
I20260810 15:20:52.841354 10479 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:52.841380 10479 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbf51c241454ec18ccdb48945c09406, State: Initialized, Role: FOLLOWER
I20260810 15:20:52.841490 10479 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf51c241454ec18ccdb48945c09406" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35107 } }
I20260810 15:20:52.841544 10479 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:52.841565 10479 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:52.841594 10479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:52.842275 10479 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf51c241454ec18ccdb48945c09406" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35107 } }
I20260810 15:20:52.842370 10479 leader_election.cc:302] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbf51c241454ec18ccdb48945c09406; no voters:
I20260810 15:20:52.842520 10479 leader_election.cc:288] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:52.842583 10484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:52.842856 10484 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [term 1 LEADER]: Becoming Leader. State: Replica: dfbf51c241454ec18ccdb48945c09406, State: Running, Role: LEADER
I20260810 15:20:52.842917 10479 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:52.843058 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10398
I20260810 15:20:52.843086 10484 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf51c241454ec18ccdb48945c09406" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35107 } }
I20260810 15:20:52.843156 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:52.843336 4353 external_mini_cluster.cc:1428] Setting key 0406a0967ddb20b6b280bdc25aae71a1
I20260810 15:20:52.843825 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbf51c241454ec18ccdb48945c09406. Latest consensus state: current_term: 1 leader_uuid: "dfbf51c241454ec18ccdb48945c09406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf51c241454ec18ccdb48945c09406" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35107 } } }
I20260810 15:20:52.843940 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:52.844602 10485 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfbf51c241454ec18ccdb48945c09406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf51c241454ec18ccdb48945c09406" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35107 } } }
I20260810 15:20:52.844709 10485 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:52.844998 10502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:52.845096 10502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:52.845268 10494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:52.845482 10494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:52.847108 10494 catalog_manager.cc:1347] Generated new cluster ID: c07c5927b2f9458693a063060c648d03
I20260810 15:20:52.847173 10494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:52.850246 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.844170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.4.64.126:35107)
0810 15:20:52.844377 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.844386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.844425 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:20:52.844703 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.844712 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.844790 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.844987 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:52.845260 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.846706 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.847807 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.848521 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.848661 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.849420 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:52.849426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:52.849426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:52.849699 (+ 273us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:52.849745 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:52.849762 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:52.849847 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:52.849917 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 15:20:52.850005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260810 15:20:52.851511 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:52.857075 10494 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:52.857877 10494 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:52.867414 10494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfbf51c241454ec18ccdb48945c09406: Generated new TSK 0
I20260810 15:20:52.867648 10494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:52.880225 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:52.876075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.4.64.126:35107)
0810 15:20:52.876146 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:52.876148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:52.876241 (+ 93us) server_negotiation.cc:407] Connection header received
0810 15:20:52.876288 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:52.876290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:52.876303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:52.876388 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:20:52.876540 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.877332 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.878372 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.878683 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.878756 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.878948 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:20:52.879999 (+ 1051us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 15:20:52.880146 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 15:20:52.880445 10511 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:52.875961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35107 (local address 127.0.0.1:52030)
0810 15:20:52.876169 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:52.876179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:52.876237 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:52.876407 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:52.876408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:52.876409 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:20:52.876470 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:52.876473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.877441 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.877442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:52.878274 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:52.878278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:52.878771 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:52.878772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:52.878825 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:52.878830 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:20:52.880212 (+ 1382us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 15:20:52.880328 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.64.126:35107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":119,"threads_started":1}
W20260810 15:20:52.880560 10498 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260810 15:20:52.880857 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10398
E20260810 15:20:52.899206 10512 webserver.cc:533] Webserver: sslize failed:
E20260810 15:20:52.899302 10512 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T15:20:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T15:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:52Z Disabled control of system clock
I20260810 15:20:52.936486 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44789
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:36849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44789
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/testchainca.cert with env {}
W20260810 15:20:53.022106 10516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:53.022324 10516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:53.022367 10516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:53.023671 10516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:53.023725 10516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:53.023737 10516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:53.023751 10516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:53.023762 10516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:20:53.023851 10516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:20:53.023883 10516 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 15:20:53.025527 10516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:53.025828 10516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:53.026087 10516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:53.031466 10522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:53.031523 10516 server_base.cc:1034] running on GCE node
W20260810 15:20:53.031489 10521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:53.031479 10524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:53.032019 10516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:53.032332 10516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:53.033483 10516 hybrid_clock.cc:648] HybridClock initialized: now 1786375253033469 us; error 37 us; skew 500 ppm
I20260810 15:20:53.034920 10516 webserver.cc:460] Webserver started at http://127.4.64.126:35265/ using document root <none> and password file <none>
I20260810 15:20:53.035203 10516 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:53.035279 10516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:53.035449 10516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:53.036552 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "3c76920863f94e93b6acbf0df1b46dfd"
format_stamp: "Formatted at 2026-08-10 15:20:53 on dist-test-slave-9zdj"
server_key: "f0174004331816d6fc1a8d26b0dd00c0"
server_key_iv: "628bea1f9b2448f4126012ff8efe88f4"
server_key_version: "encryptionkey@0"
I20260810 15:20:53.036955 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "3c76920863f94e93b6acbf0df1b46dfd"
format_stamp: "Formatted at 2026-08-10 15:20:53 on dist-test-slave-9zdj"
server_key: "f0174004331816d6fc1a8d26b0dd00c0"
server_key_iv: "628bea1f9b2448f4126012ff8efe88f4"
server_key_version: "encryptionkey@0"
I20260810 15:20:53.038496 10516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:53.039331 10530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:53.039539 10516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:53.039649 10516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "3c76920863f94e93b6acbf0df1b46dfd"
format_stamp: "Formatted at 2026-08-10 15:20:53 on dist-test-slave-9zdj"
server_key: "f0174004331816d6fc1a8d26b0dd00c0"
server_key_iv: "628bea1f9b2448f4126012ff8efe88f4"
server_key_version: "encryptionkey@0"
I20260810 15:20:53.039726 10516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:53.060750 10516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:53.061120 10516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:53.061280 10516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:53.065573 10516 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44789
I20260810 15:20:53.065649 10594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44789 every 8 connection(s)
I20260810 15:20:53.066051 10516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:53.067111 10595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:53.069482 10595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd: Bootstrap starting.
I20260810 15:20:53.070139 10595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:53.070436 10595 log.cc:824] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:53.071211 10595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd: No bootstrap required, opened a new log
I20260810 15:20:53.071730 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10516
I20260810 15:20:53.071847 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:53.072052 4353 external_mini_cluster.cc:1428] Setting key da3d6a2e19323cfcd630a70c9af72aea
I20260810 15:20:53.073662 10595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c76920863f94e93b6acbf0df1b46dfd" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44789 } }
I20260810 15:20:53.073884 10595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:53.073938 10595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c76920863f94e93b6acbf0df1b46dfd, State: Initialized, Role: FOLLOWER
I20260810 15:20:53.074082 10595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c76920863f94e93b6acbf0df1b46dfd" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44789 } }
I20260810 15:20:53.074172 10595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:53.074204 10595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:53.074267 10595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:53.075047 10595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c76920863f94e93b6acbf0df1b46dfd" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44789 } }
I20260810 15:20:53.075155 10595 leader_election.cc:302] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c76920863f94e93b6acbf0df1b46dfd; no voters:
I20260810 15:20:53.075316 10595 leader_election.cc:288] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:53.075399 10602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:53.075559 10595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:53.075565 10602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [term 1 LEADER]: Becoming Leader. State: Replica: 3c76920863f94e93b6acbf0df1b46dfd, State: Running, Role: LEADER
I20260810 15:20:53.075700 10602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c76920863f94e93b6acbf0df1b46dfd" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44789 } }
I20260810 15:20:53.076120 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c76920863f94e93b6acbf0df1b46dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c76920863f94e93b6acbf0df1b46dfd" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44789 } } }
I20260810 15:20:53.076347 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:53.076112 10602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c76920863f94e93b6acbf0df1b46dfd. Latest consensus state: current_term: 1 leader_uuid: "3c76920863f94e93b6acbf0df1b46dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c76920863f94e93b6acbf0df1b46dfd" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44789 } } }
I20260810 15:20:53.076591 10602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:53.076740 10611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:53.077641 10611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:53.079438 10611 catalog_manager.cc:1347] Generated new cluster ID: 57d765d0cc6f4b87ad1d85eceeb4c892
I20260810 15:20:53.079514 10611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:53.079587 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:53.072617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.4.64.126:44789)
0810 15:20:53.072857 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:53.072867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:53.072903 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:20:53.073557 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:53.073567 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:53.073689 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:53.073858 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:53.074142 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:53.075926 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:53.077182 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:53.077687 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:53.077882 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:53.078597 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:53.078600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:53.078601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:53.078791 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:53.078861 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:53.078877 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:53.078953 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:53.079207 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 15:20:53.079304 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260810 15:20:53.080562 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:53.087294 10611 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:53.088076 10611 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:53.097710 10611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c76920863f94e93b6acbf0df1b46dfd: Generated new TSK 0
I20260810 15:20:53.097940 10611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:53.234942 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:53.107771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.4.64.126:44789)
0810 15:20:53.107827 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:53.107829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:53.107938 (+ 109us) server_negotiation.cc:407] Connection header received
0810 15:20:53.108041 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:53.108042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:53.108054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:53.108123 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:20:53.108295 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:53.108788 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:53.110087 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:53.110393 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:53.110471 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:53.110699 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:20:53.234608 (+123909us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 15:20:53.234876 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44382: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 15:20:53.235023 10601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44382: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 15:20:53.235123 10628 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:53.107672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44789 (local address 127.0.0.1:44382)
0810 15:20:53.107851 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:53.107864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:53.107943 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:53.108140 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:53.108141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:53.108143 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:20:53.108221 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:53.108225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:53.108898 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:53.108899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:53.109953 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:53.109957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:53.110492 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:53.110493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:53.110560 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:53.110566 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:20:53.234832 (+124266us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 15:20:53.234977 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.64.126:44789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260810 15:20:53.235657 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10516
2026-08-10T15:20:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (330 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (99 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (103 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:53 dist-test-slave-9zdj krb5kdc[10641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:53 dist-test-slave-9zdj krb5kdc[10641](info): set up 2 sockets
Aug 10 15:20:53 dist-test-slave-9zdj krb5kdc[10641](info): commencing operation
krb5kdc: starting...
W20260810 15:20:54.665158 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:54 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:20:54.801867 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:33133
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:33133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:54.888744 10655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:54.889021 10655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:54.889065 10655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:54.890307 10655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:54.890383 10655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:54.890408 10655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:54.890430 10655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:54.890458 10655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:54.891969 10655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:33133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:54.892261 10655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:54.892498 10655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:54.898058 10655 server_base.cc:1034] running on GCE node
W20260810 15:20:54.898016 10661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:54.898011 10662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:54.898007 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:54.898693 10655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:54.899034 10655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:54.900279 10655 hybrid_clock.cc:648] HybridClock initialized: now 1786375254900273 us; error 44 us; skew 500 ppm
Aug 10 15:20:54 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375254, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:54.903100 10655 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:20:54.903708 10655 webserver.cc:460] Webserver started at http://127.4.64.126:38937/ using document root <none> and password file <none>
I20260810 15:20:54.903952 10655 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:54.904021 10655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:54.904188 10655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:54.905354 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "53b7e6c9017048f9bda25d69e617a1d5"
format_stamp: "Formatted at 2026-08-10 15:20:54 on dist-test-slave-9zdj"
server_key: "2ce97d50606f47fe9deba5bf4c4f3514"
server_key_iv: "97a69e5130fd0affe25422ede986eb2f"
server_key_version: "encryptionkey@0"
I20260810 15:20:54.905782 10655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "53b7e6c9017048f9bda25d69e617a1d5"
format_stamp: "Formatted at 2026-08-10 15:20:54 on dist-test-slave-9zdj"
server_key: "2ce97d50606f47fe9deba5bf4c4f3514"
server_key_iv: "97a69e5130fd0affe25422ede986eb2f"
server_key_version: "encryptionkey@0"
I20260810 15:20:54.907333 10655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:54.908126 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:54.908401 10655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:54.908501 10655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "53b7e6c9017048f9bda25d69e617a1d5"
format_stamp: "Formatted at 2026-08-10 15:20:54 on dist-test-slave-9zdj"
server_key: "2ce97d50606f47fe9deba5bf4c4f3514"
server_key_iv: "97a69e5130fd0affe25422ede986eb2f"
server_key_version: "encryptionkey@0"
I20260810 15:20:54.908576 10655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:54.923041 10655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:54.926064 10655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:54.926270 10655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:54.930900 10655 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:33133
I20260810 15:20:54.930989 10730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:33133 every 8 connection(s)
I20260810 15:20:54.931384 10655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:54.932469 10731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:54.934921 10731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5: Bootstrap starting.
I20260810 15:20:54.935586 10731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:54.935863 10731 log.cc:824] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:54.936571 10731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5: No bootstrap required, opened a new log
I20260810 15:20:54.938305 10731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b7e6c9017048f9bda25d69e617a1d5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33133 } }
I20260810 15:20:54.938474 10731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:54.938519 10731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b7e6c9017048f9bda25d69e617a1d5, State: Initialized, Role: FOLLOWER
I20260810 15:20:54.938700 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10655
I20260810 15:20:54.938694 10731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b7e6c9017048f9bda25d69e617a1d5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33133 } }
I20260810 15:20:54.938781 10731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:54.938807 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:54.938823 10731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:54.938869 10731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:54.939056 4353 external_mini_cluster.cc:1428] Setting key 06c3577a4a456dd4b7c18f9566651f3e
I20260810 15:20:54.939635 10731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b7e6c9017048f9bda25d69e617a1d5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33133 } }
I20260810 15:20:54.939761 10731 leader_election.cc:302] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53b7e6c9017048f9bda25d69e617a1d5; no voters:
I20260810 15:20:54.939953 10731 leader_election.cc:288] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:54.940208 10736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:54.940238 10731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:54.940363 10736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [term 1 LEADER]: Becoming Leader. State: Replica: 53b7e6c9017048f9bda25d69e617a1d5, State: Running, Role: LEADER
I20260810 15:20:54.940456 10736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b7e6c9017048f9bda25d69e617a1d5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33133 } }
I20260810 15:20:54.940814 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53b7e6c9017048f9bda25d69e617a1d5. Latest consensus state: current_term: 1 leader_uuid: "53b7e6c9017048f9bda25d69e617a1d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b7e6c9017048f9bda25d69e617a1d5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33133 } } }
I20260810 15:20:54.940896 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:54.941066 10737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53b7e6c9017048f9bda25d69e617a1d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53b7e6c9017048f9bda25d69e617a1d5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 33133 } } }
I20260810 15:20:54.941131 10742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:54.941151 10737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:54.941883 10742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:54.943856 10742 catalog_manager.cc:1347] Generated new cluster ID: b49c0348685e419aa88a5fda9be12d7a
I20260810 15:20:54.943924 10742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:20:54 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:54.952298 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:54.939612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.4.64.126:33133)
0810 15:20:54.939928 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:54.939940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:54.939975 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:20:54.940108 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:54.940116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:54.940174 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:54.941269 (+ 1095us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:54.941390 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:54.942838 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:54.943815 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:54.944441 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:54.944557 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:54.947483 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:54.947488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:54.947488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:54.947502 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:54.950472 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:54.950897 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:54.950901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:54.950901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:54.951035 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:54.951255 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:54.951256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:54.951256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:54.951622 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:54.951802 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:54.951876 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 15:20:54.952032 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":120,"threads_started":1}
I20260810 15:20:54.956558 10742 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:54.957064 10742 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:54.962497 10742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53b7e6c9017048f9bda25d69e617a1d5: Generated new TSK 0
I20260810 15:20:54.962718 10742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:20:54.996822 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:33133
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:55.095156 10760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:55.095494 10760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:55.095541 10760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:55.097213 10760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:55.097285 10760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:55.097337 10760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:55.099123 10760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:55.099444 10760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:55.099702 10760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:55.105662 10760 server_base.cc:1034] running on GCE node
W20260810 15:20:55.105630 10769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:55.105630 10767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:55.105610 10766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:55.106124 10760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:55.106410 10760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:55.107570 10760 hybrid_clock.cc:648] HybridClock initialized: now 1786375255107535 us; error 59 us; skew 500 ppm
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.110509 10760 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:20:55.111025 10760 webserver.cc:460] Webserver started at http://127.4.64.65:39193/ using document root <none> and password file <none>
I20260810 15:20:55.111294 10760 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:55.111351 10760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:55.111534 10760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:55.112735 10760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "47b52f4479f24478b5729722aa7af299"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "7917df2ab3350df5cb1811bb07d6a997"
server_key_iv: "953bd01c43da2906236e69c1afedee8d"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.113132 10760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "47b52f4479f24478b5729722aa7af299"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "7917df2ab3350df5cb1811bb07d6a997"
server_key_iv: "953bd01c43da2906236e69c1afedee8d"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.114750 10760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:55.115681 10776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.115906 10760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:55.116001 10760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "47b52f4479f24478b5729722aa7af299"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "7917df2ab3350df5cb1811bb07d6a997"
server_key_iv: "953bd01c43da2906236e69c1afedee8d"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.116065 10760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:55.133106 10760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:55.135499 10760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:55.135628 10760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:55.135862 10760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:55.136210 10760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:55.136246 10760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.136271 10760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:55.136320 10760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.142704 10760 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:37227
I20260810 15:20:55.142776 10907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:37227 every 8 connection(s)
I20260810 15:20:55.143139 10760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:55.143770 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10760
I20260810 15:20:55.143901 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:55.144146 4353 external_mini_cluster.cc:1428] Setting key 533df500991f27dfe1323b912dfc83bd
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
I20260810 15:20:55.154054 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.144383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:60491 (local address 127.4.64.126:33133)
0810 15:20:55.144445 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.144447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.144458 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:55.145691 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.145692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.145709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.145759 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.146440 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.146980 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.148030 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.148268 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.148355 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.150693 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:55.150695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:55.150695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:55.150709 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:55.152338 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.152882 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.152883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.152884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.152946 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.153246 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.153247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.153247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.153413 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:55.153484 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:55.153843 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 15:20:55.153943 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:20:55.154461 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.144060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:33133 (local address 127.4.64.65:60491)
0810 15:20:55.144259 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:55.144331 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:55.145521 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:55.145834 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:55.145838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:55.146122 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:55.146338 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:55.146342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.147099 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.147100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:55.147936 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:55.147937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.148369 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.148370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:55.148500 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.149084 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:55.149093 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:55.150593 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:55.152542 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:55.152550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:55.152552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:55.152793 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:55.153029 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:55.153030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:55.153031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:55.153167 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:55.153487 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:55.153495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:55.153694 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 15:20:55.153818 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 15:20:55.153990 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260810 15:20:55.155006 10908 heartbeater.cc:347] Connected to a master server at 127.4.64.126:33133
I20260810 15:20:55.155118 10908 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:55.155303 10908 heartbeater.cc:511] Master 127.4.64.126:33133 requested a full tablet report, sending...
I20260810 15:20:55.155864 10693 ts_manager.cc:194] Registered new tserver with Master: 47b52f4479f24478b5729722aa7af299 (127.4.64.65:37227)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
I20260810 15:20:55.156611 10693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:60491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:20:55.173803 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:33133
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:55.263566 10915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:55.263797 10915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:55.263823 10915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:55.264950 10915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:55.264987 10915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:55.265013 10915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:55.266582 10915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:55.266805 10915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:55.267000 10915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:55.272548 10920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:55.272557 10923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:55.272568 10921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:55.272629 10915 server_base.cc:1034] running on GCE node
I20260810 15:20:55.272984 10915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:55.273204 10915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:55.274391 10915 hybrid_clock.cc:648] HybridClock initialized: now 1786375255274374 us; error 26 us; skew 500 ppm
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.277465 10915 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:20:55.277988 10915 webserver.cc:460] Webserver started at http://127.4.64.66:33969/ using document root <none> and password file <none>
I20260810 15:20:55.278242 10915 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:55.278317 10915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:55.278472 10915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:55.279601 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "0e43348acf5c419f9e0435b91e369f15"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "9563898b2e8dd298471d75ca40604c29"
server_key_iv: "f9a8fbc2dc4be34843436adf8c75d26f"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.279971 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "0e43348acf5c419f9e0435b91e369f15"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "9563898b2e8dd298471d75ca40604c29"
server_key_iv: "f9a8fbc2dc4be34843436adf8c75d26f"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.281374 10915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:55.282239 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.282470 10915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:55.282575 10915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "0e43348acf5c419f9e0435b91e369f15"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "9563898b2e8dd298471d75ca40604c29"
server_key_iv: "f9a8fbc2dc4be34843436adf8c75d26f"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.282653 10915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:55.297782 10915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:55.300403 10915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:55.300577 10915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:55.300853 10915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:55.301186 10915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:55.301254 10915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.301288 10915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:55.301316 10915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.307492 10915 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:36491
I20260810 15:20:55.307562 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:36491 every 8 connection(s)
I20260810 15:20:55.307909 10915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:55.310915 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 10915
I20260810 15:20:55.311028 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:55.311228 4353 external_mini_cluster.cc:1428] Setting key bf49a3a104a7f8b26d375fe06a4a6603
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:55.318367 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.308751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:36247 (local address 127.4.64.126:33133)
0810 15:20:55.308811 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.308813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.308987 (+ 174us) server_negotiation.cc:407] Connection header received
0810 15:20:55.309939 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.309941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.309952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.310016 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.310684 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.311533 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.312699 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.312898 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.312971 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.315502 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:55.315504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:55.315504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:55.315512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:55.316907 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.317341 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.317343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.317343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.317406 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.317693 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.317694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.317694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.317816 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:55.317885 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:55.318177 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 15:20:55.318273 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:20:55.318552 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.308673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:33133 (local address 127.4.64.66:36247)
0810 15:20:55.308839 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:55.308883 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:55.309828 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:55.310093 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:55.310096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:55.310371 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:55.310572 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:55.310576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.311641 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.311642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:55.312599 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:55.312602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.312981 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.312982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:55.313111 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.313688 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:55.313697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:55.315398 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:55.317011 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:55.317019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:55.317028 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:55.317265 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:55.317471 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:55.317472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:55.317472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:55.317586 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:55.317889 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:55.317897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:55.318042 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 15:20:55.318143 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 15:20:55.318309 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260810 15:20:55.319139 11061 heartbeater.cc:347] Connected to a master server at 127.4.64.126:33133
I20260810 15:20:55.319264 11061 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:55.319458 11061 heartbeater.cc:511] Master 127.4.64.126:33133 requested a full tablet report, sending...
I20260810 15:20:55.319900 10693 ts_manager.cc:194] Registered new tserver with Master: 0e43348acf5c419f9e0435b91e369f15 (127.4.64.66:36491)
I20260810 15:20:55.320382 10693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:36247
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:20:55.341733 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:33133
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:55.429690 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:55.429921 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:55.429947 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:55.431164 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:55.431208 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:55.431236 11067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:55.432729 11067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:55.432934 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:55.433139 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:55.438817 11077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:55.439006 11067 server_base.cc:1034] running on GCE node
W20260810 15:20:55.438810 11074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:55.438819 11075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:55.439358 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:55.439672 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:55.440852 11067 hybrid_clock.cc:648] HybridClock initialized: now 1786375255440824 us; error 39 us; skew 500 ppm
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.443938 11067 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:20:55.444505 11067 webserver.cc:460] Webserver started at http://127.4.64.67:39597/ using document root <none> and password file <none>
I20260810 15:20:55.444748 11067 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:55.444819 11067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:55.444972 11067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:55.446177 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "1425688dbf0c49968466c926da9590ec"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "9887a45e29047094f024c8b575918a40"
server_key_iv: "a00795674ce157f5a051199e9a39ea88"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.446561 11067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "1425688dbf0c49968466c926da9590ec"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "9887a45e29047094f024c8b575918a40"
server_key_iv: "a00795674ce157f5a051199e9a39ea88"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.447921 11067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:55.448782 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.449059 11067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:55.449245 11067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "1425688dbf0c49968466c926da9590ec"
format_stamp: "Formatted at 2026-08-10 15:20:55 on dist-test-slave-9zdj"
server_key: "9887a45e29047094f024c8b575918a40"
server_key_iv: "a00795674ce157f5a051199e9a39ea88"
server_key_version: "encryptionkey@0"
I20260810 15:20:55.449342 11067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:55.486796 11067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:55.489269 11067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:55.489441 11067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:55.489727 11067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:55.490065 11067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:55.490116 11067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.490152 11067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:55.490203 11067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:55.495806 11067 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:41279
I20260810 15:20:55.495910 11212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:41279 every 8 connection(s)
I20260810 15:20:55.496220 11067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:55.499003 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 11067
I20260810 15:20:55.499204 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:55.499403 4353 external_mini_cluster.cc:1428] Setting key b2ad8e74032e5abeda0ee29f5fbba06a
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:20:55.506340 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.496983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:43295 (local address 127.4.64.126:33133)
0810 15:20:55.497042 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.497044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.497323 (+ 279us) server_negotiation.cc:407] Connection header received
0810 15:20:55.498342 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.498343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.498355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.498418 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.499111 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.499834 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.500987 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.501205 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.501289 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.503551 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:55.503553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:55.503553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:55.503560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:55.504979 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.505419 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.505420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.505420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.505481 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.505772 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.505773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.505774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.505884 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:55.505940 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:55.506195 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 15:20:55.506273 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:20:55.506489 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.496958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:33133 (local address 127.4.64.67:43295)
0810 15:20:55.497139 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:55.497182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:55.498234 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:55.498489 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:55.498493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:55.498792 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:55.498999 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:55.499004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.499934 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.499936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:55.500894 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:55.500897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.501302 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.501303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:55.501437 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.502032 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:55.502041 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:55.503464 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:55.505068 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:55.505074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:55.505076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:55.505326 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:55.505549 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:55.505549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:55.505550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:55.505722 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:55.505941 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:55.505946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:55.506069 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 15:20:55.506148 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 15:20:55.506284 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260810 15:20:55.506999 11213 heartbeater.cc:347] Connected to a master server at 127.4.64.126:33133
I20260810 15:20:55.507117 11213 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:55.507324 11213 heartbeater.cc:511] Master 127.4.64.126:33133 requested a full tablet report, sending...
I20260810 15:20:55.507728 10693 ts_manager.cc:194] Registered new tserver with Master: 1425688dbf0c49968466c926da9590ec (127.4.64.67:41279)
I20260810 15:20:55.508108 10693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:43295
I20260810 15:20:55.511564 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/test-admin.keytab.
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.530718 4353 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 15:20:55.532617 11222 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:55.531685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.4.64.65:37227)
0810 15:20:55.531874 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.531879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.531890 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:55.531918 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.531919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.531934 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.532001 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.532525 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46834: BlockingRecv error: recv got EOF from 127.0.0.1:46834 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
W20260810 15:20:55.532704 4353 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc' of type FILE
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.533836 4353 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.534893 4353 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:55.543678 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.535507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.4.64.65:37227)
0810 15:20:55.535553 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.535555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.535585 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:20:55.535666 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.535667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.535688 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.535751 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.536180 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.536960 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.537775 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.538119 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.538201 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.540478 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:55.540486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:55.540487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:55.540494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:55.542305 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.542732 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.542735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.542735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.542796 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.542968 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.542969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.542969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.543107 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:55.543276 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:55.543526 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 15:20:55.543595 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 15:20:55.545599 11222 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:55.544985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.4.64.65:37227)
0810 15:20:55.545046 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.545048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.545111 (+ 63us) server_negotiation.cc:407] Connection header received
0810 15:20:55.545202 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.545202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.545214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.545272 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.545548 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46844: BlockingRecv error: recv got EOF from 127.0.0.1:46844 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 15:20:55.545754 4353 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc' of type FILE
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.546800 4353 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.547684 4353 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:55.555325 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.548242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46856 (local address 127.4.64.65:37227)
0810 15:20:55.548338 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.548340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.548348 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:55.548365 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.548366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.548374 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.548422 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.548795 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.549291 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.550036 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.550236 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.550329 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.552628 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:55.552629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:55.552630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:55.552636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:55.554178 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.554551 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.554553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.554553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.554611 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.554809 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.554810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.554810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.554918 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:55.555014 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:55.555137 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:20:55.555223 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 15:20:55.556857 11222 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:55.556318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.4.64.65:37227)
0810 15:20:55.556352 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.556354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.556417 (+ 63us) server_negotiation.cc:407] Connection header received
0810 15:20:55.556459 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.556459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.556466 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.556508 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.556802 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46870: BlockingRecv error: recv got EOF from 127.0.0.1:46870 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 15:20:55.556937 4353 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786375228510355-4353-0/krb5kdc/krb5cc' of type FILE
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.557955 4353 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:55.559010 4353 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[10641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:20:55.566936 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:55.559618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.4.64.65:37227)
0810 15:20:55.559663 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:55.559665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:55.559831 (+ 166us) server_negotiation.cc:407] Connection header received
0810 15:20:55.559885 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:55.559886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:55.559894 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:55.559945 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:55.560384 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.560909 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.561900 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:55.562088 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:55.562153 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:55.564272 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:55.564274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:55.564274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:55.564280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:55.565874 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.566202 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.566203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.566204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.566257 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:55.566450 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:55.566451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:55.566451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:55.566579 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:55.566696 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:55.566799 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 15:20:55.566864 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:20:55.567762 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10760
I20260810 15:20:55.576596 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10915
I20260810 15:20:55.582752 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 11067
I20260810 15:20:55.591215 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 10655
2026-08-10T15:20:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2153 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[11237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[11237](info): set up 2 sockets
Aug 10 15:20:55 dist-test-slave-9zdj krb5kdc[11237](info): commencing operation
krb5kdc: starting...
W20260810 15:20:56.719391 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:20:56 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab.
Entry for principal oryx/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab.
I20260810 15:20:56.861280 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:41729
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab
--principal=oryx/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:41729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:56.948683 11253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:56.948915 11253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:56.948939 11253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:56.950138 11253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:56.950196 11253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:56.950210 11253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:56.950222 11253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:56.950234 11253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:56.951771 11253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:41729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab
--principal=oryx/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:56.952003 11253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:56.952211 11253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:56.957376 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:56.957448 11260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:56.957444 11261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:56.957657 11253 server_base.cc:1034] running on GCE node
I20260810 15:20:56.957945 11253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:56.958259 11253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:56.959445 11253 hybrid_clock.cc:648] HybridClock initialized: now 1786375256959437 us; error 37 us; skew 500 ppm
Aug 10 15:20:56 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:56.962173 11253 init.cc:377] Logged in from keytab as oryx/127.4.64.126@KRBTEST.COM (short username oryx)
I20260810 15:20:56.962649 11253 webserver.cc:460] Webserver started at http://127.4.64.126:42993/ using document root <none> and password file <none>
I20260810 15:20:56.962929 11253 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:56.963001 11253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:56.963150 11253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:56.964257 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0"
format_stamp: "Formatted at 2026-08-10 15:20:56 on dist-test-slave-9zdj"
server_key: "e8e1aaede76c60b162f2e5939b9a2eb5"
server_key_iv: "349cf237af79bf9f054398e960e06529"
server_key_version: "encryptionkey@0"
I20260810 15:20:56.964632 11253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0"
format_stamp: "Formatted at 2026-08-10 15:20:56 on dist-test-slave-9zdj"
server_key: "e8e1aaede76c60b162f2e5939b9a2eb5"
server_key_iv: "349cf237af79bf9f054398e960e06529"
server_key_version: "encryptionkey@0"
I20260810 15:20:56.966120 11253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:20:56.966962 11271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:56.967186 11253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:56.967314 11253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0"
format_stamp: "Formatted at 2026-08-10 15:20:56 on dist-test-slave-9zdj"
server_key: "e8e1aaede76c60b162f2e5939b9a2eb5"
server_key_iv: "349cf237af79bf9f054398e960e06529"
server_key_version: "encryptionkey@0"
I20260810 15:20:56.967408 11253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:56.979439 11253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:56.981804 11253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:56.981976 11253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:56.986657 11253 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:41729
I20260810 15:20:56.986704 11326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:41729 every 8 connection(s)
I20260810 15:20:56.987093 11253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:56.987577 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 11253
I20260810 15:20:56.987706 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:20:56.987907 4353 external_mini_cluster.cc:1428] Setting key c2cb80c7cd464a9b48d8cfb9b1b0049f
I20260810 15:20:56.988510 11327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:56.992419 11327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Bootstrap starting.
I20260810 15:20:56.993132 11327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:56.993518 11327 log.cc:824] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:56.994445 11327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: No bootstrap required, opened a new log
Aug 10 15:20:56 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:20:56.996587 11327 raft_consensus.cc:348] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } }
I20260810 15:20:56.996793 11327 raft_consensus.cc:374] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:56.996865 11327 raft_consensus.cc:729] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b412a7829a2d4a5eb3c9c7b8d977cdf0, State: Initialized, Role: FOLLOWER
I20260810 15:20:56.997015 11327 consensus_queue.cc:260] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } }
I20260810 15:20:56.997143 11327 raft_consensus.cc:388] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:56.997187 11327 raft_consensus.cc:482] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:56.997246 11327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:56.998208 11327 raft_consensus.cc:504] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } }
I20260810 15:20:56.998342 11327 leader_election.cc:302] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b412a7829a2d4a5eb3c9c7b8d977cdf0; no voters:
I20260810 15:20:56.998534 11327 leader_election.cc:288] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:56.998626 11332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:56.998906 11327 sys_catalog.cc:564] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:56.999007 11332 raft_consensus.cc:686] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 LEADER]: Becoming Leader. State: Replica: b412a7829a2d4a5eb3c9c7b8d977cdf0, State: Running, Role: LEADER
I20260810 15:20:56.999177 11332 consensus_queue.cc:237] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } }
I20260810 15:20:56.999622 11335 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } }
I20260810 15:20:56.999608 11332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b412a7829a2d4a5eb3c9c7b8d977cdf0. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } }
I20260810 15:20:56.999720 11335 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:56.999725 11332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:57.000224 11343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:57.001477 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:56.988661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.4.64.126:41729)
0810 15:20:56.988888 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:56.988899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:56.988937 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:20:56.989825 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:56.989836 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:56.989906 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:56.990079 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:56.990721 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:56.992302 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:56.993269 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:56.993964 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:56.994082 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:56.996775 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:56.996780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:56.996780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:56.996796 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:56.999271 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:56.999822 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:56.999826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:56.999827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:56.999950 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.000221 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.000222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.000223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.000631 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.000813 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.001053 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.001220 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
I20260810 15:20:57.001688 11343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:57.003418 11343 catalog_manager.cc:1347] Generated new cluster ID: c8e6b649a7064ed891dd7928bb9254c0
I20260810 15:20:57.003485 11343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:57.022053 11343 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:57.022922 11343 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:57.032006 11343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Generated new TSK 0
I20260810 15:20:57.032253 11343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab.
Entry for principal oryx/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab.
I20260810 15:20:57.074118 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:41729
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:57.168654 11360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:57.168944 11360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:57.168993 11360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:57.170461 11360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:57.170538 11360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:57.170590 11360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:57.172578 11360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:57.172921 11360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:57.173206 11360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:57.179216 11370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.179200 11366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.179337 11360 server_base.cc:1034] running on GCE node
W20260810 15:20:57.179198 11367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.179674 11360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:57.179991 11360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:57.181176 11360 hybrid_clock.cc:648] HybridClock initialized: now 1786375257181159 us; error 56 us; skew 500 ppm
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:57.184039 11360 init.cc:377] Logged in from keytab as oryx/127.4.64.65@KRBTEST.COM (short username oryx)
I20260810 15:20:57.184610 11360 webserver.cc:460] Webserver started at http://127.4.64.65:37259/ using document root <none> and password file <none>
I20260810 15:20:57.184899 11360 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:57.184973 11360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:57.185145 11360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:57.186448 11360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "a3cc093ec09b0fda2c31b975cf556323"
server_key_iv: "c0d76d0416eccc88067b46c4e92c2d0e"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.186849 11360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "a3cc093ec09b0fda2c31b975cf556323"
server_key_iv: "c0d76d0416eccc88067b46c4e92c2d0e"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.188616 11360 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:20:57.189555 11377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.189901 11360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:57.190025 11360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "a3cc093ec09b0fda2c31b975cf556323"
server_key_iv: "c0d76d0416eccc88067b46c4e92c2d0e"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.190104 11360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:57.201821 11360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:57.204378 11360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:57.204586 11360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:57.204909 11360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:57.205317 11360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:57.205379 11360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.205416 11360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:57.205448 11360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.212664 11360 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:45551
I20260810 15:20:57.212726 11503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:45551 every 8 connection(s)
I20260810 15:20:57.213124 11360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:20:57.221352 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 11360
I20260810 15:20:57.221480 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:20:57.221731 4353 external_mini_cluster.cc:1428] Setting key 89e62314eab125f0061b935fe57f4909
I20260810 15:20:57.223904 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.214024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:51419 (local address 127.4.64.126:41729)
0810 15:20:57.214111 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.214114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.214340 (+ 226us) server_negotiation.cc:407] Connection header received
0810 15:20:57.214874 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.214875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.214886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.214953 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.216011 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.216563 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.217897 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.218118 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.218179 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.220559 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.220561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.220561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.220569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.222349 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.222839 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.222841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.222841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.222922 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.223207 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.223208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.223208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.223333 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.223410 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.223699 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.223808 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 15:20:57.224182 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.213994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:51419)
0810 15:20:57.214188 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.214238 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.214783 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.215013 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.215017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.215687 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.215911 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.215917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.216670 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.216671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.217808 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.217811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.218230 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.218231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.218345 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.218878 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.218887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.220444 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.222467 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.222475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.222478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.222748 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.223003 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.223004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.223004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.223123 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.223401 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.223409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.223558 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.223663 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.223845 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260810 15:20:57.224870 11504 heartbeater.cc:347] Connected to a master server at 127.4.64.126:41729
I20260810 15:20:57.224984 11504 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:57.225209 11504 heartbeater.cc:511] Master 127.4.64.126:41729 requested a full tablet report, sending...
I20260810 15:20:57.225840 11289 ts_manager.cc:194] Registered new tserver with Master: bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551)
I20260810 15:20:57.226527 11289 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:51419
WARNING: no policy specified for oryx/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab.
Entry for principal oryx/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab.
I20260810 15:20:57.253459 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:41729
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:57.358211 11509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:57.358464 11509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:57.358505 11509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:57.359702 11509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:57.359772 11509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:57.359819 11509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:57.361380 11509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:57.361732 11509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:57.361991 11509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:57.367637 11517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.367623 11515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.367604 11514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.367946 11509 server_base.cc:1034] running on GCE node
I20260810 15:20:57.368201 11509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:57.368453 11509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:57.369684 11509 hybrid_clock.cc:648] HybridClock initialized: now 1786375257369650 us; error 52 us; skew 500 ppm
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:57.372763 11509 init.cc:377] Logged in from keytab as oryx/127.4.64.66@KRBTEST.COM (short username oryx)
I20260810 15:20:57.373212 11509 webserver.cc:460] Webserver started at http://127.4.64.66:42769/ using document root <none> and password file <none>
I20260810 15:20:57.373436 11509 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:57.373486 11509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:57.373585 11509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:57.374711 11509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "30ee1bc0f4634e9db852e812838ff817"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "2706f67929c0e1f99546a983046d6721"
server_key_iv: "83dafc03e88ebc6476d822e884851df6"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.375104 11509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "30ee1bc0f4634e9db852e812838ff817"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "2706f67929c0e1f99546a983046d6721"
server_key_iv: "83dafc03e88ebc6476d822e884851df6"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.376521 11509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:57.377410 11524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.377702 11509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:57.377818 11509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "30ee1bc0f4634e9db852e812838ff817"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "2706f67929c0e1f99546a983046d6721"
server_key_iv: "83dafc03e88ebc6476d822e884851df6"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.377905 11509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:57.387310 11509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:57.390239 11509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:57.390408 11509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:57.390650 11509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:57.390977 11509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:57.391012 11509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.391052 11509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:57.391078 11509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.397185 11509 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38915
I20260810 15:20:57.397346 11652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38915 every 8 connection(s)
I20260810 15:20:57.397662 11509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:57.400688 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 11509
I20260810 15:20:57.400807 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:20:57.401019 4353 external_mini_cluster.cc:1428] Setting key 0d2cdc5303eacbd3bf6c83a92e474d0b
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:20:57.408955 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.398495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:36735 (local address 127.4.64.126:41729)
0810 15:20:57.398584 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.398586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.398859 (+ 273us) server_negotiation.cc:407] Connection header received
0810 15:20:57.399502 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.399503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.399517 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.399582 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.400742 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.401399 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.402584 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.402913 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.402992 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.405691 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.405692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.405692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.405700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.407211 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.407754 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.407755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.407755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.407814 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.408205 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.408211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.408212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.408335 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.408390 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.408803 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.408886 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 15:20:57.409284 11656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.398468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:36735)
0810 15:20:57.398717 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.398759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.399378 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.399659 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.399664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.400327 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.400631 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.400636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.401492 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.401493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.402493 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.402496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.403022 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.403023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.403252 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.403894 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.403904 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.405522 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.407297 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.407306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.407309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.407670 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.407887 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.407889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.407889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.408060 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.408408 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.408416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.408624 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.408764 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.408984 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":135,"threads_started":1}
I20260810 15:20:57.409948 11654 heartbeater.cc:347] Connected to a master server at 127.4.64.126:41729
I20260810 15:20:57.410079 11654 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:57.410316 11654 heartbeater.cc:511] Master 127.4.64.126:41729 requested a full tablet report, sending...
I20260810 15:20:57.410815 11289 ts_manager.cc:194] Registered new tserver with Master: 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915)
I20260810 15:20:57.411235 11289 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:36735
WARNING: no policy specified for oryx/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab.
Entry for principal oryx/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab.
I20260810 15:20:57.434473 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:41729
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:57.520583 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:57.520831 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:57.520875 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:57.522106 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:57.522167 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:57.522209 11660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:57.523734 11660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:57.524013 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:57.524263 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:57.529645 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.529805 11660 server_base.cc:1034] running on GCE node
W20260810 15:20:57.529686 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.529673 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.530150 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:57.530411 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:57.531584 11660 hybrid_clock.cc:648] HybridClock initialized: now 1786375257531563 us; error 29 us; skew 500 ppm
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:57.534355 11660 init.cc:377] Logged in from keytab as oryx/127.4.64.67@KRBTEST.COM (short username oryx)
I20260810 15:20:57.534802 11660 webserver.cc:460] Webserver started at http://127.4.64.67:40469/ using document root <none> and password file <none>
I20260810 15:20:57.535032 11660 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:57.535076 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:57.535171 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:57.536269 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "d14f9b1454274be79c7893fe538ba840"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "da32eda0665f66bf4da7cb4374b42163"
server_key_iv: "0bde924a8294d6f2a77391474d60480e"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.536582 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "d14f9b1454274be79c7893fe538ba840"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "da32eda0665f66bf4da7cb4374b42163"
server_key_iv: "0bde924a8294d6f2a77391474d60480e"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.537982 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:57.538815 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.538990 11660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:57.539098 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "d14f9b1454274be79c7893fe538ba840"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "da32eda0665f66bf4da7cb4374b42163"
server_key_iv: "0bde924a8294d6f2a77391474d60480e"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.539176 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:57.554745 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:57.557045 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:57.557245 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:57.557509 11660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:57.557906 11660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:57.557964 11660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.558023 11660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:57.558053 11660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.564168 11660 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43981
I20260810 15:20:57.564229 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43981 every 8 connection(s)
I20260810 15:20:57.564586 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:20:57.570861 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 11660
I20260810 15:20:57.570957 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:20:57.571192 4353 external_mini_cluster.cc:1428] Setting key f018c78a4c754c95678de1695e9e0b49
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:20:57.574913 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.565538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:60839 (local address 127.4.64.126:41729)
0810 15:20:57.565675 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.565678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.565879 (+ 201us) server_negotiation.cc:407] Connection header received
0810 15:20:57.566380 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.566382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.566393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.566459 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.567440 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.567974 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.569011 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.569209 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.569270 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.572133 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.572135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.572135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.572143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.573535 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.574013 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.574014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.574015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.574078 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.574327 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.574328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.574328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.574447 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.574500 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.574769 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.574847 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:20:57.575026 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.565483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:60839)
0810 15:20:57.565745 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.565790 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.566289 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.566535 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.566539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.567137 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.567345 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.567349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.568065 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.568066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.568927 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.568929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.569293 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.569294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.569404 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.570041 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.570049 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.572010 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.573672 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.573679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.573681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.573929 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.574153 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.574153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.574153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.574262 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.574495 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.574498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.574617 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.574692 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.574820 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":180,"threads_started":1}
I20260810 15:20:57.575621 11808 heartbeater.cc:347] Connected to a master server at 127.4.64.126:41729
I20260810 15:20:57.575733 11808 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:57.575944 11808 heartbeater.cc:511] Master 127.4.64.126:41729 requested a full tablet report, sending...
I20260810 15:20:57.576378 11289 ts_manager.cc:194] Registered new tserver with Master: d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981)
I20260810 15:20:57.576731 11289 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:60839
I20260810 15:20:57.583432 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.64.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.125.keytab.
Entry for principal oryx/127.4.64.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.125.keytab.
I20260810 15:20:57.613547 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:34561
--webserver_interface=127.4.64.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.125.keytab
--principal=oryx/127.4.64.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:41729,127.4.64.125:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:57.701608 11813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:57.701870 11813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:57.701910 11813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:57.703119 11813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:57.703208 11813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:57.703250 11813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:57.703293 11813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:57.703315 11813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:57.704854 11813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:41729,127.4.64.125:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.125.keytab
--principal=oryx/127.4.64.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.64.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
W20260810 15:20:57.705005 11813 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.64.126:41729,127.4.64.125:34561'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 15:20:57.705190 11813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:57.705438 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:57.710602 11821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.710642 11819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.710621 11818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.710837 11813 server_base.cc:1034] running on GCE node
I20260810 15:20:57.711086 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:57.711370 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:57.712523 11813 hybrid_clock.cc:648] HybridClock initialized: now 1786375257712492 us; error 60 us; skew 500 ppm
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:57.715113 11813 init.cc:377] Logged in from keytab as oryx/127.4.64.125@KRBTEST.COM (short username oryx)
I20260810 15:20:57.715586 11813 webserver.cc:460] Webserver started at http://127.4.64.125:37899/ using document root <none> and password file <none>
I20260810 15:20:57.715776 11813 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:57.715819 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:57.715911 11813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:57.716957 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/instance:
uuid: "a0009246e17642e4aeec69a80efed348"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "4f923327934b97f92bee817aa91f6dba"
server_key_iv: "4888a72621d728a606ffa34831f0749c"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.717275 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal/instance:
uuid: "a0009246e17642e4aeec69a80efed348"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "4f923327934b97f92bee817aa91f6dba"
server_key_iv: "4888a72621d728a606ffa34831f0749c"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.718714 11813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:57.719503 11828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.719699 11813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:57.719790 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
uuid: "a0009246e17642e4aeec69a80efed348"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "4f923327934b97f92bee817aa91f6dba"
server_key_iv: "4888a72621d728a606ffa34831f0749c"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.719866 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:57.740120 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:57.742383 11813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:57.742570 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:57.746856 11813 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.125:34561
I20260810 15:20:57.746912 11890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.125:34561 every 8 connection(s)
I20260810 15:20:57.747315 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
I20260810 15:20:57.748432 11891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:57.750015 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 11813
I20260810 15:20:57.750108 11891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.125@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
I20260810 15:20:57.760241 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.750957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.4.64.126:41729)
0810 15:20:57.751018 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.751020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.751192 (+ 172us) server_negotiation.cc:407] Connection header received
0810 15:20:57.751667 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.751668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.751683 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.751740 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.752419 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.752947 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.753955 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.754179 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.754264 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.756755 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.756757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.756757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.756766 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.758470 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.758950 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.758951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.758951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.759012 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.759342 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.759344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.759344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.759594 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.759695 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.760070 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.760158 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:20:57.760511 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.750925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39188)
0810 15:20:57.751083 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.751116 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.751560 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.751767 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.751770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.752045 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.752309 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.752315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.753050 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.753051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.753843 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.753846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.754282 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.754283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.754389 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.754910 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.754920 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.756638 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.758558 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.758563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.758565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.758863 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.759078 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.759079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.759080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.759234 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.759689 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.759695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.759894 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.760037 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.760228 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260810 15:20:57.761168 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.750802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.4.64.125:34561)
0810 15:20:57.751059 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.751069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.751099 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:20:57.751545 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.751552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.751572 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.751694 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.752347 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.753148 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.753962 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.754358 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.754435 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.757261 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.757271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.757271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.757278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.759147 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.759695 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.759698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.759698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.759769 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.759977 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.759978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.759978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.760252 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.760966 (+ 714us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.761011 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.761079 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260810 15:20:57.761328 11891 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.125@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
I20260810 15:20:57.767917 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.761687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.4.64.125:34561)
0810 15:20:57.761740 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.761742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.761783 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:20:57.761850 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.761851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.761864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.761920 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.762204 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.762685 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.763428 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.763621 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.763685 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.765501 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.765502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.765502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.765508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.767006 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.767313 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.767314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.767314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.767368 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.767528 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.767529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.767529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.767628 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.767684 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.767791 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.767856 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:57.768105 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.761619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51768)
0810 15:20:57.761732 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.761744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.761799 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.761928 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.761929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.762052 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.762096 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.762097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.762775 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.762776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.763326 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.763328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.763713 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.763714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.763810 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.764174 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.764178 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.765423 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.767081 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.767082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.767082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.767247 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.767426 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.767427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.767427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.767479 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.767684 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.767684 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.767719 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.767775 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.767904 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":46}
I20260810 15:20:57.769691 11891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Bootstrap starting.
I20260810 15:20:57.770659 11891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:57.770952 11891 log.cc:824] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:57.771734 11891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: No bootstrap required, opened a new log
I20260810 15:20:57.773185 11891 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } }
I20260810 15:20:57.773319 11891 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:57.773342 11891 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0009246e17642e4aeec69a80efed348, State: Initialized, Role: FOLLOWER
I20260810 15:20:57.773464 11891 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } }
I20260810 15:20:57.773757 11900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } } }
I20260810 15:20:57.773809 11891 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:57.773865 11900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:20:57.775812 11913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:57.775878 11913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:57.777851 11813 master_runner.cc:428] Detected that this master a0009246e17642e4aeec69a80efed348 is joining an existing cluster
I20260810 15:20:57.777927 11813 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.64.125:34561
I20260810 15:20:57.787720 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.779250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.4.64.126:41729)
0810 15:20:57.779295 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.779298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.779650 (+ 352us) server_negotiation.cc:407] Connection header received
0810 15:20:57.779688 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.779689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.779701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.779764 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.780280 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.780969 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.782356 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.782680 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.782759 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.784054 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.784055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.784055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.784062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.786057 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.786798 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.786799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.786799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.786892 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.787108 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.787109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.787109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.787267 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.787340 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.787535 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.787620 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:57.788305 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.779348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39198)
0810 15:20:57.779538 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.779553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.779629 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.779817 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.779818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.780070 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.780180 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.780182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.781054 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.781056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.782259 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.782261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.782936 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.782937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.783002 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.783514 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.783519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.783959 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.786269 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.786271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.786271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.786689 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.786970 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.786971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.786971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.787031 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.787372 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.787373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.787418 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.787974 (+ 556us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.788087 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260810 15:20:57.788416 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.779279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.4.64.125:34561)
0810 15:20:57.779764 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.779766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.779834 (+ 68us) server_negotiation.cc:407] Connection header received
0810 15:20:57.779892 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.779893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.779903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.779955 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.780341 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.781099 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.782822 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.783112 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.783183 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.784767 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.784768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.784769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.784776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.786826 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.787340 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.787341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.787342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.787405 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.787605 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.787606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.787606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.787716 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.787809 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.788119 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.788267 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":466}
I20260810 15:20:57.788697 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.779616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51784)
0810 15:20:57.779768 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.779778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.779831 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.779958 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.779958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.780192 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.780272 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.780274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.781180 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.781181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.782259 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.782261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.783510 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.783511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.783586 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.784291 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.784295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.784675 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.786911 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.786912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.786912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.787208 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.787475 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.787476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.787477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.787537 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.787963 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.787964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.788000 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.788092 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.788205 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":41,"thread_start_us":100,"threads_started":1}
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.126@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
I20260810 15:20:57.799433 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.791496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.4.64.125:34561)
0810 15:20:57.791545 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.791547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.791603 (+ 56us) server_negotiation.cc:407] Connection header received
0810 15:20:57.791720 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.791721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.791729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.791780 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.792307 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.792856 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.794089 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.794295 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.794364 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.796634 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.796636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.796636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.796644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.798067 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.798503 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.798504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.798505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.798566 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.798801 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.798801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.798802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.798917 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.798987 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.799189 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.799332 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:20:57.799467 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.791322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51796)
0810 15:20:57.791497 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.791544 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.791624 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.791793 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.791796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.792146 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.792209 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.792211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.792939 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.792940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.793995 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.793998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.794375 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.794376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.794463 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.794902 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.794908 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.796535 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.798171 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.798180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.798182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.798408 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.798662 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.798663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.798664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.798730 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.798992 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.799001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.799059 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.799149 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.799328 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260810 15:20:57.800153 11289 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.64.125:34561: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a0009246e17642e4aeec69a80efed348"
member_type: NON_VOTER
last_known_addr {
host: "127.4.64.125"
port: 34561
}
attrs {
promote: true
}
}
dest_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0"
cas_config_opid_index: -1
I20260810 15:20:57.800484 11289 consensus_queue.cc:237] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } }
I20260810 15:20:57.800971 11924 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a0009246e17642e4aeec69a80efed348 (127.4.64.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } }
I20260810 15:20:57.801381 11864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:57.802026 11924 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.64.125:34561
I20260810 15:20:57.802330 11923 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } }
I20260810 15:20:57.802380 11864 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Refusing update from remote peer b412a7829a2d4a5eb3c9c7b8d977cdf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 15:20:57.802423 11923 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:57.802366 11813 master.cc:537] Master@127.4.64.125:34561 shutting down...
I20260810 15:20:57.802706 11923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:20:57.803475 11274 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 15:20:57.804111 11274 consensus_peers.cc:487] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 -> Peer a0009246e17642e4aeec69a80efed348 (127.4.64.125:34561): Couldn't send request to peer a0009246e17642e4aeec69a80efed348. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 15:20:57.805357 11813 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:57.805456 11813 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:57.805507 11813 tablet_replica.cc:330] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: stopping tablet replica
I20260810 15:20:57.819252 11813 master.cc:552] Master@127.4.64.125:34561 shutdown complete.
I20260810 15:20:57.819325 11813 master_runner.cc:305] Clearing existing system tablet
I20260810 15:20:57.819468 11813 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:57.819522 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:57.820303 11813 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:57.820881 11928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.821035 11813 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:57.821100 11813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:57.821136 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
uuid: "a0009246e17642e4aeec69a80efed348"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "4f923327934b97f92bee817aa91f6dba"
server_key_iv: "4888a72621d728a606ffa34831f0749c"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.822185 11813 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:57.823886 11813 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 15:20:57.823968 11813 log.cc:1196] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 15:20:57.824183 11813 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Deleting consensus metadata
I20260810 15:20:57.824312 11813 master_runner.cc:315] Copying system tablet from 127.4.64.126:41729
I20260810 15:20:57.824975 11813 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: tablet copy: Beginning tablet copy session from remote peer at address 127.4.64.126:41729
I20260810 15:20:57.832110 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.825483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.4.64.126:41729)
0810 15:20:57.825540 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.825542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.825603 (+ 61us) server_negotiation.cc:407] Connection header received
0810 15:20:57.825713 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.825714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.825728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.825799 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.826217 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.826764 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.827860 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.828054 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.828123 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.829282 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.829284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.829284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.829292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.830852 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.831273 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.831274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.831274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.831336 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.831564 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.831565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.831565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.831711 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.831782 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.831957 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.832034 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:20:57.832163 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.825374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39202)
0810 15:20:57.825514 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:57.825526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:57.825604 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:57.825814 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:57.825815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:57.826030 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:57.826104 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.826106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.826845 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.826846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.827764 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:57.827767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.828149 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.828150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:57.828225 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.828726 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:57.828731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:57.829146 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:57.830941 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.830942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.830942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.831192 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.831403 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:57.831404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:57.831405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:57.831477 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:57.831785 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:57.831786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:57.831838 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 15:20:57.831921 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:20:57.832037 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":92,"threads_started":1}
I20260810 15:20:57.832921 11309 tablet_copy_service.cc:140] P b412a7829a2d4a5eb3c9c7b8d977cdf0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a0009246e17642e4aeec69a80efed348 ({username='oryx', principal='oryx/127.4.64.125@KRBTEST.COM'} at 127.0.0.1:39202)
I20260810 15:20:57.833038 11309 tablet_copy_service.cc:161] P b412a7829a2d4a5eb3c9c7b8d977cdf0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a0009246e17642e4aeec69a80efed348 at {username='oryx', principal='oryx/127.4.64.125@KRBTEST.COM'} at 127.0.0.1:39202: session id = a0009246e17642e4aeec69a80efed348-00000000000000000000000000000000
I20260810 15:20:57.833880 11309 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 15:20:57.834684 11813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 15:20:57.836122 11813 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: tablet copy: Starting download of 0 data blocks...
I20260810 15:20:57.836292 11813 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: tablet copy: Starting download of 1 WAL segments...
I20260810 15:20:57.837297 11813 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 15:20:57.838410 11309 tablet_copy_service.cc:342] P b412a7829a2d4a5eb3c9c7b8d977cdf0: Request end of tablet copy session a0009246e17642e4aeec69a80efed348-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.64.125@KRBTEST.COM'} at 127.0.0.1:39202
I20260810 15:20:57.838477 11309 tablet_copy_service.cc:433] P b412a7829a2d4a5eb3c9c7b8d977cdf0: ending tablet copy session a0009246e17642e4aeec69a80efed348-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a0009246e17642e4aeec69a80efed348
I20260810 15:20:57.841296 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:57.842814 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.842922 11813 server_base.cc:1034] running on GCE node
W20260810 15:20:57.842811 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:57.842839 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:57.843138 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:57.843410 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:57.844548 11813 hybrid_clock.cc:648] HybridClock initialized: now 1786375257844533 us; error 31 us; skew 500 ppm
Aug 10 15:20:57 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375257, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:57.846100 11813 init.cc:377] Logged in from keytab as oryx/127.4.64.125@KRBTEST.COM (short username oryx)
I20260810 15:20:57.846448 11813 webserver.cc:460] Webserver started at http://127.4.64.125:33899/ using document root <none> and password file <none>
I20260810 15:20:57.846570 11813 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:57.846607 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:57.847373 11813 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:57.847978 11947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:57.848111 11813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:57.848176 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
uuid: "a0009246e17642e4aeec69a80efed348"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "4f923327934b97f92bee817aa91f6dba"
server_key_iv: "4888a72621d728a606ffa34831f0749c"
server_key_version: "encryptionkey@0"
I20260810 15:20:57.848229 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:57.861285 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:57.863142 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:57.865852 11813 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.125:34561
I20260810 15:20:57.866093 12009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.125:34561 every 8 connection(s)
I20260810 15:20:57.866257 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
I20260810 15:20:57.867449 12010 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:20:57.868131 12010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Bootstrap starting.
I20260810 15:20:57.871524 12010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:20:57.871806 12010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Bootstrap complete.
I20260810 15:20:57.872071 12010 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } }
I20260810 15:20:57.872177 12010 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a0009246e17642e4aeec69a80efed348, State: Initialized, Role: LEARNER
I20260810 15:20:57.872222 12010 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } }
I20260810 15:20:57.872418 12010 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:57.872435 12013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } }
I20260810 15:20:57.872570 12013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:20:57.874038 12024 catalog_manager.cc:1259] Loaded cluster ID: c8e6b649a7064ed891dd7928bb9254c0
I20260810 15:20:57.874089 12024 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: loading cluster ID for follower catalog manager: success
I20260810 15:20:57.875154 12024 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: acquiring CA information for follower catalog manager: success
I20260810 15:20:57.875424 12024 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:20:57.920601 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:57.913675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.4.64.125:34561)
0810 15:20:57.913859 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:57.913861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:57.913869 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:57.913895 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:57.913895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:57.913905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:57.913987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:57.914652 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.915413 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.916209 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:57.916428 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:57.916501 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:57.917931 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:57.917933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:57.917933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:57.917946 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:57.919373 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.919759 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.919760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.919761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.919832 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:57.920036 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:57.920037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:57.920037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:57.920155 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:57.920263 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:57.920364 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 15:20:57.920457 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.4.64.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.124.keytab.
Entry for principal oryx/127.4.64.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.124.keytab.
I20260810 15:20:57.952834 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:40099
--webserver_interface=127.4.64.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.124.keytab
--principal=oryx/127.4.64.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:58.038813 12028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:58.039029 12028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:58.039053 12028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:58.040273 12028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:58.040324 12028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:58.040339 12028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:58.040352 12028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:58.040365 12028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:58.042212 12028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.124.keytab
--principal=oryx/127.4.64.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.64.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:58.042497 12028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:58.042759 12028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:58.048070 12034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.048079 12035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:58.048285 12028 server_base.cc:1034] running on GCE node
W20260810 15:20:58.048091 12040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:58.048652 12028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:58.048964 12028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:58.050125 12028 hybrid_clock.cc:648] HybridClock initialized: now 1786375258050097 us; error 58 us; skew 500 ppm
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:58.053095 12028 init.cc:377] Logged in from keytab as oryx/127.4.64.124@KRBTEST.COM (short username oryx)
I20260810 15:20:58.053603 12028 webserver.cc:460] Webserver started at http://127.4.64.124:34135/ using document root <none> and password file <none>
I20260810 15:20:58.053885 12028 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:58.053929 12028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:58.054025 12028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:58.055122 12028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/instance:
uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
format_stamp: "Formatted at 2026-08-10 15:20:58 on dist-test-slave-9zdj"
server_key: "23a7c6872df429443a2e9635e54886ec"
server_key_iv: "f271eeccf59c5a03771031b05bd0033f"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.055451 12028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal/instance:
uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
format_stamp: "Formatted at 2026-08-10 15:20:58 on dist-test-slave-9zdj"
server_key: "23a7c6872df429443a2e9635e54886ec"
server_key_iv: "f271eeccf59c5a03771031b05bd0033f"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.056957 12028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:58.057813 12049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.058019 12028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:58.058130 12028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
format_stamp: "Formatted at 2026-08-10 15:20:58 on dist-test-slave-9zdj"
server_key: "23a7c6872df429443a2e9635e54886ec"
server_key_iv: "f271eeccf59c5a03771031b05bd0033f"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.058223 12028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:58.070592 12028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:58.073143 12028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:58.073314 12028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:58.077595 12028 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.124:40099
I20260810 15:20:58.077744 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.124:40099 every 8 connection(s)
I20260810 15:20:58.078070 12028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
I20260810 15:20:58.078958 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 12028
I20260810 15:20:58.079156 12112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:58.081843 12112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.64.124@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.124@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:20:58.090939 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.082599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.4.64.126:41729)
0810 15:20:58.082653 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.082655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.082727 (+ 72us) server_negotiation.cc:407] Connection header received
0810 15:20:58.082802 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.082803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.082812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.082859 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.083452 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.084203 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.085493 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.085779 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.085836 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.088070 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.088071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.088071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.088080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.089598 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.090064 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.090065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.090066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.090131 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.090323 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.090324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.090324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.090472 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.090544 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.090760 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.090844 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:20:58.091042 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.079637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.4.64.124:40099)
0810 15:20:58.079828 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.079836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.079861 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:20:58.080284 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.080291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.080382 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.080505 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.081338 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.082536 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.083610 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.084291 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.084386 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.087126 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.087133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.087133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.087139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.089194 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.089567 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.089571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.089572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.089715 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.089946 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.089947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.089947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.090335 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.090502 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.090589 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.090791 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260810 15:20:58.091049 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.082463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39210)
0810 15:20:58.082605 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.082644 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.082741 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.082899 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.082902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.083283 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.083369 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.083371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.084311 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.084312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.085396 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.085400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.085860 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.085861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.085954 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.086439 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.086447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.087980 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.089698 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.089704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.089706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.089987 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.090208 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.090209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.090209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.090263 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.090545 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.090551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.090601 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.090694 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.090866 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":29,"thread_start_us":88,"threads_started":1}
I20260810 15:20:58.091919 12112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.124@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
I20260810 15:20:58.099248 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.092386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.4.64.125:34561)
0810 15:20:58.092446 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.092448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.092457 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:20:58.092486 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.092486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.092504 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.092551 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.092839 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.093512 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.094362 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.094574 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.094636 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.096424 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.096426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.096426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.096434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.098056 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.098432 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.098433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.098433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.098498 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.098739 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.098740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.098740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.098856 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.098960 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.099093 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.099180 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:20:58.099249 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.092275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51822)
0810 15:20:58.092323 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.092334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.092376 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.092561 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.092562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.092676 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.092733 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.092735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.093602 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.093603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.094173 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.094175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.094649 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.094650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.094709 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.095109 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.095114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.096342 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.098154 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.098155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.098155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.098340 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.098596 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.098597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.098598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.098679 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.098964 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.098965 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.099012 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.099070 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.099164 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:20:58.099769 12112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.124@KRBTEST.COM for oryx/127.4.64.124@KRBTEST.COM
I20260810 15:20:58.107175 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.100219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58746)
0810 15:20:58.100276 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.100287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.100332 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.100474 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.100475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.100611 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.100647 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.100648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.101440 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.101441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.101973 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.101975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.102488 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.102489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.102545 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.102981 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.102985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.104240 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.106125 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.106126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.106127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.106362 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.106561 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.106562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.106562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.106625 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.106910 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.106911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.106960 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.107018 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.107101 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:20:58.107255 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.100232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.4.64.124:40099)
0810 15:20:58.100276 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.100277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.100320 (+ 43us) server_negotiation.cc:407] Connection header received
0810 15:20:58.100382 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.100383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.100391 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.100431 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.100751 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.101358 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.102066 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.102401 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.102466 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.104346 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.104347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.104348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.104354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.106021 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.106440 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.106441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.106441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.106498 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.106683 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.106684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.106684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.106823 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.106893 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.107078 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.107165 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:58.108745 12112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Bootstrap starting.
I20260810 15:20:58.109795 12112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:58.110054 12112 log.cc:824] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:58.110924 12112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: No bootstrap required, opened a new log
I20260810 15:20:58.112488 12112 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } }
I20260810 15:20:58.112664 12112 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:58.112692 12112 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb9e1aaf0ce46a681870b85b75b5fbe, State: Initialized, Role: FOLLOWER
I20260810 15:20:58.112844 12112 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } }
I20260810 15:20:58.113132 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } } }
I20260810 15:20:58.113195 12112 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:58.113245 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:20:58.115053 12131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 15:20:58.115113 12119 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:58.115213 12119 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } }
W20260810 15:20:58.115123 12131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:58.115468 12119 leader_election.cc:288] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), a0009246e17642e4aeec69a80efed348 (127.4.64.125:34561)
I20260810 15:20:58.115672 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0009246e17642e4aeec69a80efed348" is_pre_election: true
I20260810 15:20:58.115803 11979 raft_consensus.cc:1799] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Handling vote request from an unknown peer ffb9e1aaf0ce46a681870b85b75b5fbe
I20260810 15:20:58.115861 11979 raft_consensus.cc:2396] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate ffb9e1aaf0ce46a681870b85b75b5fbe for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 15:20:58.115865 11299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" is_pre_election: true
I20260810 15:20:58.116357 12053 leader_election.cc:302] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffb9e1aaf0ce46a681870b85b75b5fbe; no voters: a0009246e17642e4aeec69a80efed348, b412a7829a2d4a5eb3c9c7b8d977cdf0
I20260810 15:20:58.116536 12119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:58.118077 12119 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 15:20:58.118593 12028 master_runner.cc:428] Detected that this master ffb9e1aaf0ce46a681870b85b75b5fbe is joining an existing cluster
I20260810 15:20:58.118660 12028 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.64.124:40099
I20260810 15:20:58.127254 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.119981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.4.64.124:40099)
0810 15:20:58.120020 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.120022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.120181 (+ 159us) server_negotiation.cc:407] Connection header received
0810 15:20:58.120272 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.120273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.120283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.120313 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.120885 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.121673 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.122889 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.123189 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.123232 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.124441 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.124443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.124443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.124450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.125962 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.126461 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.126462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.126463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.126521 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.126725 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.126726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.126727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.126839 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.126912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.127051 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.127145 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:58.127343 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.119994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58752)
0810 15:20:58.120129 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.120139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.120204 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.120370 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.120371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.120713 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.120797 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.120799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.121765 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.121766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.122793 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.122795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.123334 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.123334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.123396 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.123940 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.123946 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.124322 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.126055 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.126056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.126057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.126230 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.126586 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.126586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.126587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.126644 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.126903 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.126904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.126946 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.127028 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.127209 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":217,"thread_start_us":74,"threads_started":1}
I20260810 15:20:58.142829 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 11360
I20260810 15:20:58.153700 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.119826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.4.64.125:34561)
0810 15:20:58.119918 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.119919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.120181 (+ 262us) server_negotiation.cc:407] Connection header received
0810 15:20:58.120276 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.120277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.120289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.120321 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.120894 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.121673 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.122892 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.123195 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.123232 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.150389 (+ 27157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.150391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.150391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.150402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.152233 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.152714 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.152715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.152716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.152780 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.152980 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.152981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.152982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.153144 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.153189 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.153421 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.153521 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 15:20:58.153837 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.119774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.4.64.126:41729)
0810 15:20:58.119881 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.119884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.120061 (+ 177us) server_negotiation.cc:407] Connection header received
0810 15:20:58.120170 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.120171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.120184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.120216 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.120822 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.121664 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.122919 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.123263 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.123319 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.150361 (+ 27042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.150363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.150364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.150374 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.152541 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.152966 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.152967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.152967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.153027 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.153224 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.153225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.153225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.153374 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.153416 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.153649 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.153738 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 15:20:58.154110 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.119794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51830)
0810 15:20:58.120113 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.120123 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.120204 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.120388 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.120389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.120714 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.120797 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.120799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.121765 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.121765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.122792 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.122795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.123332 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.123332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.123396 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.149668 (+ 26272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.149675 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.150250 (+ 575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.152356 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.152357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.152357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.152607 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.152852 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.152852 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.152853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.152916 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.153236 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.153237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.153305 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.153802 (+ 497us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.153938 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":90,"thread_start_us":100,"threads_started":1}
I20260810 15:20:58.155145 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:45551
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:41729
--webserver_port=37259
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:20:58.155156 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.119722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39220)
0810 15:20:58.119985 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.119998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.120091 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.120301 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.120302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.120638 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.120719 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.120722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.121765 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.121766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.122809 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.122812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.123418 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.123419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.123499 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.149577 (+ 26078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.149585 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.150183 (+ 598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.152649 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.152650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.152650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.152874 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.153096 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.153097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.153097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.153153 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.153465 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.153466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.153508 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.154919 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.155041 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375256, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.126@KRBTEST.COM for oryx/127.4.64.124@KRBTEST.COM
I20260810 15:20:58.167349 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.158937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58756)
0810 15:20:58.158994 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.159004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.159054 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.159212 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.159213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.159407 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.159465 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.159467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.160450 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.160452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.161329 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.161332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.162023 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.162024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.162103 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.162578 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.162583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.164183 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.166178 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.166179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.166180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.166436 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.166647 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.166648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.166648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.166706 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.166991 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.166992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.167040 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.167104 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.167226 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:20:58.167737 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.158998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.4.64.124:40099)
0810 15:20:58.159041 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.159043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.159053 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:58.159125 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.159126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.159136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.159198 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.159551 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.160343 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.161417 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.161939 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.162024 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.164301 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.164303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.164303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.164311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.166075 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.166521 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.166522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.166523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.166582 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.166770 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.166770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.166771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.166906 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.166965 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.167557 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.167642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:58.168334 11289 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.64.124:40099: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
member_type: NON_VOTER
last_known_addr {
host: "127.4.64.124"
port: 40099
}
attrs {
promote: true
}
}
dest_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0"
cas_config_opid_index: 5
I20260810 15:20:58.168632 11289 consensus_queue.cc:237] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } }
I20260810 15:20:58.169116 11923 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.170310 11923 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.64.124:40099
I20260810 15:20:58.170398 11923 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.170471 11923 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.170841 12028 master.cc:537] Master@127.4.64.124:40099 shutting down...
I20260810 15:20:58.171178 12081 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Refusing update from remote peer b412a7829a2d4a5eb3c9c7b8d977cdf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 15:20:58.171499 11924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:20:58.172478 11276 consensus_peers.cc:487] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 -> Peer ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124:40099): Couldn't send request to peer ffb9e1aaf0ce46a681870b85b75b5fbe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 15:20:58.175028 12028 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:58.175132 12028 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:58.175179 12028 tablet_replica.cc:330] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: stopping tablet replica
I20260810 15:20:58.178386 12028 master.cc:552] Master@127.4.64.124:40099 shutdown complete.
I20260810 15:20:58.178442 12028 master_runner.cc:305] Clearing existing system tablet
I20260810 15:20:58.178579 12028 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:58.178630 12028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:58.179404 12028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:20:58.179613 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.169155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.4.64.125:34561)
0810 15:20:58.169206 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.169208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.172990 (+ 3782us) server_negotiation.cc:407] Connection header received
0810 15:20:58.173028 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.173030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.173060 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.173141 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.173450 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.174286 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.176280 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.177074 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.177458 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.177700 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.179470 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.179534 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:58.179840 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.169074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51834)
0810 15:20:58.170946 (+ 1872us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.170958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.171021 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.173179 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.173181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.173183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.173236 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.173238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.174463 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.174464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.176149 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.176154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.179261 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.179262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.179347 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.179381 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.179713 (+ 332us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.179769 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851}
I20260810 15:20:58.180276 11979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Refusing update from remote peer b412a7829a2d4a5eb3c9c7b8d977cdf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 15:20:58.180655 11924 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 15:20:58.181324 11979 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.182890 12013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b412a7829a2d4a5eb3c9c7b8d977cdf0. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.182974 12013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:20:58.183010 12013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: NON_VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: true } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.183048 12013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:20:58.183396 11923 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: attempting to promote NON_VOTER a0009246e17642e4aeec69a80efed348 to VOTER
I20260810 15:20:58.183632 11923 consensus_queue.cc:237] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } }
I20260810 15:20:58.183802 12146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.183928 12028 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:58.183969 12028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.183990 12028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
format_stamp: "Formatted at 2026-08-10 15:20:58 on dist-test-slave-9zdj"
server_key: "23a7c6872df429443a2e9635e54886ec"
server_key_iv: "f271eeccf59c5a03771031b05bd0033f"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.184376 11979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 LEARNER]: Refusing update from remote peer b412a7829a2d4a5eb3c9c7b8d977cdf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 15:20:58.184595 11923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 15:20:58.184813 12028 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:58.185071 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.184154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58772)
0810 15:20:58.184976 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.185041 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":803}
I20260810 15:20:58.185355 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.185290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58780)
0810 15:20:58.185318 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.185337 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260810 15:20:58.185405 11276 consensus_peers.cc:487] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 -> Peer ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124:40099): Couldn't send request to peer ffb9e1aaf0ce46a681870b85b75b5fbe. Status: Network error: Client connection negotiation failed: client connection to 127.4.64.124:40099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 15:20:58.185742 11924 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0009246e17642e4aeec69a80efed348 (127.4.64.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.186453 12028 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 15:20:58.186548 12028 log.cc:1196] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 15:20:58.186612 11924 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.186703 11924 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.186757 11924 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.186808 11924 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.186870 12028 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Deleting consensus metadata
I20260810 15:20:58.187016 12028 master_runner.cc:315] Copying system tablet from 127.4.64.126:41729
I20260810 15:20:58.187335 11979 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a0009246e17642e4aeec69a80efed348 (127.4.64.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.187772 12028 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: tablet copy: Beginning tablet copy session from remote peer at address 127.4.64.126:41729
I20260810 15:20:58.188413 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.188511 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:58.195891 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.188190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.4.64.126:41729)
0810 15:20:58.188230 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.188232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.188560 (+ 328us) server_negotiation.cc:407] Connection header received
0810 15:20:58.188716 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.188717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.188729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.188761 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.189190 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.190044 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.191188 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.191505 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.191549 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.192756 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.192758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.192758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.192765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.194535 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.194973 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.194975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.194975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.195035 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.195324 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.195325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.195325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.195468 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.195509 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.195703 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.195788 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:58.196496 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.188287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39236)
0810 15:20:58.188465 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.188476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.188634 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.188829 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.188830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.189048 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.189101 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.189103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.190149 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.190150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.191081 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.191084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.191637 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.191637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.191707 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.192221 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.192226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.192650 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.194642 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.194643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.194644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.194886 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.195105 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.195106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.195106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.195162 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.195553 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.195554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.195600 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.196275 (+ 675us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.196403 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260810 15:20:58.196981 11309 tablet_copy_service.cc:140] P b412a7829a2d4a5eb3c9c7b8d977cdf0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ffb9e1aaf0ce46a681870b85b75b5fbe ({username='oryx', principal='oryx/127.4.64.124@KRBTEST.COM'} at 127.0.0.1:39236)
I20260810 15:20:58.197042 11309 tablet_copy_service.cc:161] P b412a7829a2d4a5eb3c9c7b8d977cdf0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ffb9e1aaf0ce46a681870b85b75b5fbe at {username='oryx', principal='oryx/127.4.64.124@KRBTEST.COM'} at 127.0.0.1:39236: session id = ffb9e1aaf0ce46a681870b85b75b5fbe-00000000000000000000000000000000
I20260810 15:20:58.197834 11309 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 15:20:58.198431 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 15:20:58.199903 12028 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: tablet copy: Starting download of 0 data blocks...
I20260810 15:20:58.200057 12028 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: tablet copy: Starting download of 1 WAL segments...
I20260810 15:20:58.200987 12028 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 15:20:58.201994 11309 tablet_copy_service.cc:342] P b412a7829a2d4a5eb3c9c7b8d977cdf0: Request end of tablet copy session ffb9e1aaf0ce46a681870b85b75b5fbe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.64.124@KRBTEST.COM'} at 127.0.0.1:39236
I20260810 15:20:58.202059 11309 tablet_copy_service.cc:433] P b412a7829a2d4a5eb3c9c7b8d977cdf0: ending tablet copy session ffb9e1aaf0ce46a681870b85b75b5fbe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ffb9e1aaf0ce46a681870b85b75b5fbe
I20260810 15:20:58.204720 12028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:58.205991 12157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.206221 12160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.206429 12158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:58.210767 12028 server_base.cc:1034] running on GCE node
I20260810 15:20:58.210896 12028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:58.211042 12028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:58.212181 12028 hybrid_clock.cc:648] HybridClock initialized: now 1786375258212161 us; error 36 us; skew 500 ppm
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:58.213827 12028 init.cc:377] Logged in from keytab as oryx/127.4.64.124@KRBTEST.COM (short username oryx)
I20260810 15:20:58.214104 12028 webserver.cc:460] Webserver started at http://127.4.64.124:39309/ using document root <none> and password file <none>
I20260810 15:20:58.214226 12028 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:58.214262 12028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:58.215065 12028 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:58.215729 12167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.215914 12028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:58.215969 12028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
format_stamp: "Formatted at 2026-08-10 15:20:58 on dist-test-slave-9zdj"
server_key: "23a7c6872df429443a2e9635e54886ec"
server_key_iv: "f271eeccf59c5a03771031b05bd0033f"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.216033 12028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:58.221891 12028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:58.223774 12028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:58.226912 12028 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.124:40099
I20260810 15:20:58.227317 12028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
I20260810 15:20:58.228259 12235 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:20:58.228777 12235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Bootstrap starting.
I20260810 15:20:58.229765 12234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.124:40099 every 8 connection(s)
I20260810 15:20:58.231875 12235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:20:58.232117 12235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: Bootstrap complete.
I20260810 15:20:58.232339 12235 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } }
I20260810 15:20:58.232424 12235 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ffb9e1aaf0ce46a681870b85b75b5fbe, State: Initialized, Role: LEARNER
I20260810 15:20:58.232475 12235 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } }
I20260810 15:20:58.232646 12235 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:58.234457 12250 catalog_manager.cc:1259] Loaded cluster ID: c8e6b649a7064ed891dd7928bb9254c0
I20260810 15:20:58.234506 12250 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: loading cluster ID for follower catalog manager: success
I20260810 15:20:58.233780 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.235262 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: This master's current role is: LEARNER
I20260810 15:20:58.235883 12250 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: acquiring CA information for follower catalog manager: success
I20260810 15:20:58.236203 12250 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 15:20:58.265249 12142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:58.265491 12142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:58.265516 12142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:58.266767 12142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:58.266820 12142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:58.266849 12142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:20:58.268428 12142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:45551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=37259
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:58.268719 12142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:58.268990 12142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:58.274813 12260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.274843 12257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.274973 12258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:58.275319 12142 server_base.cc:1034] running on GCE node
I20260810 15:20:58.275557 12142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:58.275820 12142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:58.276986 12142 hybrid_clock.cc:648] HybridClock initialized: now 1786375258276964 us; error 37 us; skew 500 ppm
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:58.279986 12142 init.cc:377] Logged in from keytab as oryx/127.4.64.65@KRBTEST.COM (short username oryx)
I20260810 15:20:58.280562 12142 webserver.cc:460] Webserver started at http://127.4.64.65:37259/ using document root <none> and password file <none>
I20260810 15:20:58.280890 12142 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:58.280997 12142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:58.282749 12142 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:58.283538 12267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.283806 12142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:58.283926 12142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "a3cc093ec09b0fda2c31b975cf556323"
server_key_iv: "c0d76d0416eccc88067b46c4e92c2d0e"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.284336 12142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:58.294785 12142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:58.297144 12142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:58.297348 12142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:58.297598 12142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:58.297979 12142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:58.298031 12142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.298072 12142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:58.298097 12142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.304488 12142 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:45551
I20260810 15:20:58.304570 12397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:45551 every 8 connection(s)
I20260810 15:20:58.304934 12142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:20:58.312361 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 12142
I20260810 15:20:58.312567 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 11509
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.124@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
I20260810 15:20:58.318742 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.306269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:36175 (local address 127.4.64.126:41729)
0810 15:20:58.306313 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.306315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.306326 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:58.307223 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.307225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.307239 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.307274 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.308284 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.309075 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310217 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.310497 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310565 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.314627 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.314629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.314629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.314638 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.316775 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.317420 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.317421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.317422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.317487 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.317839 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.317840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.317841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.318037 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.318129 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.318520 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.318620 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:58.319221 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.306086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:36175)
0810 15:20:58.306229 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.306239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.307092 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.307457 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.307462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.307911 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.308158 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.308164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.309169 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.309170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.310122 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.310125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.310586 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.310734 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.311366 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.311378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.314508 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.316883 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.316892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.316895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.317323 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.317568 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.317568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.317569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.317737 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.318136 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.318143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.318345 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.318880 (+ 535us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.319007 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":126,"thread_start_us":69,"threads_started":1}
I20260810 15:20:58.319482 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.305768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:40799 (local address 127.4.64.124:40099)
0810 15:20:58.306451 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.306453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.306460 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:58.307250 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.307251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.307266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.307327 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.308274 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.308937 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310376 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.310647 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310715 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.315439 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.315441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.315441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.315450 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.317813 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.318120 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.318121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.318122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.318186 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.318381 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.318381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.318382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.318527 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.318588 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.319283 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.319378 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":62,"threads_started":1}
I20260810 15:20:58.319221 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.305775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.4.64.65:40799)
0810 15:20:58.305957 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.305999 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.307079 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.307457 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.307462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.307909 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.308158 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.308164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.309052 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.309053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.310271 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.310274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.310771 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.310831 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.311365 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.311378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.315319 (+ 3941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.317883 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.317884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.317885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.318047 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.318259 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.318260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.318260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.318315 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.318615 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.318616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.318656 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.318743 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.318960 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":114,"thread_start_us":99,"threads_started":1}
I20260810 15:20:58.319723 12399 heartbeater.cc:347] Connected to a master server at 127.4.64.126:41729
I20260810 15:20:58.319954 12399 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.320178 12399 heartbeater.cc:511] Master 127.4.64.126:41729 requested a full tablet report, sending...
I20260810 15:20:58.320297 12398 heartbeater.cc:347] Connected to a master server at 127.4.64.124:40099
I20260810 15:20:58.320330 12398 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.320387 12398 heartbeater.cc:511] Master 127.4.64.124:40099 requested a full tablet report, sending...
I20260810 15:20:58.320606 11289 ts_manager.cc:194] Re-registered known tserver with Master: bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551)
I20260810 15:20:58.320741 12188 ts_manager.cc:194] Registered new tserver with Master: bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551)
I20260810 15:20:58.321122 11289 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:36175
I20260810 15:20:58.322288 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.305757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:49463 (local address 127.4.64.125:34561)
0810 15:20:58.305804 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.305807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.306108 (+ 301us) server_negotiation.cc:407] Connection header received
0810 15:20:58.307252 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.307253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.307267 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.307301 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.308267 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.309051 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310355 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.310649 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.317405 (+ 6660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.317407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.317408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.317417 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.320047 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.321336 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.321337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.321338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.321414 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.321703 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.321704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.321705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.321862 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.321939 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.322155 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.322226 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:58.322504 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.305775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.4.64.65:49463)
0810 15:20:58.305957 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.305999 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.307081 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.307465 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.307466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.307901 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.308157 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.308162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.309133 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.309134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.310271 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.310274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.310749 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.310750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.310834 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.311505 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.311510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.317259 (+ 5749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.320921 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.320923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.320923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.321218 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.321506 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.321507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.321507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.321588 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.321951 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.321953 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.322034 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.322130 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.322292 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":82,"thread_start_us":108,"threads_started":1}
I20260810 15:20:58.322957 12400 heartbeater.cc:347] Connected to a master server at 127.4.64.125:34561
I20260810 15:20:58.323011 12400 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.323084 12400 heartbeater.cc:511] Master 127.4.64.125:34561 requested a full tablet report, sending...
I20260810 15:20:58.323531 11966 ts_manager.cc:194] Registered new tserver with Master: bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551)
I20260810 15:20:58.330848 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:38915
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:41729
--webserver_port=42769
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:58.435230 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:58.435482 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:58.435510 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:58.436800 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:58.436878 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:58.436932 12408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:20:58.440069 12408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:38915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=42769
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:58.440392 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:58.440793 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:58.448079 12408 server_base.cc:1034] running on GCE node
W20260810 15:20:58.448079 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.448079 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.448568 12418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:58.448874 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:58.449234 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:58.450402 12408 hybrid_clock.cc:648] HybridClock initialized: now 1786375258450384 us; error 38 us; skew 500 ppm
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:58.453747 12408 init.cc:377] Logged in from keytab as oryx/127.4.64.66@KRBTEST.COM (short username oryx)
I20260810 15:20:58.454275 12408 webserver.cc:460] Webserver started at http://127.4.64.66:42769/ using document root <none> and password file <none>
I20260810 15:20:58.454546 12408 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:58.454614 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:58.456207 12408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:58.457450 12425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.457765 12408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:58.457891 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "30ee1bc0f4634e9db852e812838ff817"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "2706f67929c0e1f99546a983046d6721"
server_key_iv: "83dafc03e88ebc6476d822e884851df6"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.458359 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:58.472196 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:58.475029 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:58.475303 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:58.475574 12408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:58.475916 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:58.475967 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.476009 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:58.476040 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.482304 12408 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38915
I20260810 15:20:58.482362 12555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38915 every 8 connection(s)
I20260810 15:20:58.482743 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:20:58.488041 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 12408
I20260810 15:20:58.488217 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 11660
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.124@KRBTEST.COM
I20260810 15:20:58.503314 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.483571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:50815 (local address 127.4.64.124:40099)
0810 15:20:58.483611 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.483612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.484036 (+ 424us) server_negotiation.cc:407] Connection header received
0810 15:20:58.484614 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.484616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.484628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.484701 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.485848 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.486368 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.487942 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.488386 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.488461 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.496548 (+ 8087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.496550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.496550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.496561 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.501450 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.501904 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.501905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.501905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.501962 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.502427 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.502428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.502429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.502561 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.502639 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.503004 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.503129 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:58.503504 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.483523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.4.64.66:50815)
0810 15:20:58.483965 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.483975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.484487 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.484779 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.484783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.485406 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.485725 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.485731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.486474 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.486474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.487760 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.487763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.489607 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.489608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.489720 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.490880 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.490885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.496433 (+ 5548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.501543 (+ 5110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.501544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.501545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.501816 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.502034 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.502035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.502035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.502344 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.502664 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.502664 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.502811 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.502932 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.503242 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":239,"thread_start_us":92,"threads_started":1}
I20260810 15:20:58.503834 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.483521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:56291 (local address 127.4.64.125:34561)
0810 15:20:58.483577 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.483579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.483898 (+ 319us) server_negotiation.cc:407] Connection header received
0810 15:20:58.484799 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.484801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.484813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.484868 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.485821 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.486610 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.488523 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.488858 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.488933 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.495575 (+ 6642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.495578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.495579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.495590 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.501127 (+ 5537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.501880 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.501881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.501882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.501950 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.502329 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.502330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.502331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.502528 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.502608 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.503551 (+ 943us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.503701 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:20:58.504046 12556 heartbeater.cc:347] Connected to a master server at 127.4.64.124:40099
I20260810 15:20:58.504072 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.483523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.4.64.66:56291)
0810 15:20:58.483737 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.483786 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.484485 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.484884 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.484884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.485417 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.485723 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.485729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.486696 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.486697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.487760 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.487763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.489026 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.489027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.489210 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.490121 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.490126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.495043 (+ 4917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.501250 (+ 6207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.501266 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.501269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.501775 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.502034 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.502034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.502035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.502232 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.502605 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.502614 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.502811 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.502932 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.503195 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":14,"spinlock_wait_cycles":7424,"thread_start_us":91,"threads_started":1}
I20260810 15:20:58.504184 12556 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.504436 12556 heartbeater.cc:511] Master 127.4.64.124:40099 requested a full tablet report, sending...
I20260810 15:20:58.504612 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:43981
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:41729
--webserver_port=40469
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:20:58.504987 12558 heartbeater.cc:347] Connected to a master server at 127.4.64.125:34561
I20260810 15:20:58.505069 12558 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.505173 12558 heartbeater.cc:511] Master 127.4.64.125:34561 requested a full tablet report, sending...
I20260810 15:20:58.505657 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.483843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:40297)
0810 15:20:58.484677 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.484687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.484740 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.485324 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.485325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.485518 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.485725 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.485731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.487857 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.487858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.488593 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.488596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.489074 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.489075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.489205 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.489971 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.489979 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.495432 (+ 5453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.504211 (+ 8779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.504212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.504213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.504450 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.504742 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.504743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.504743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.504823 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.505182 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.505183 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.505300 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.505392 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.505519 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":57,"thread_start_us":75,"threads_started":1}
I20260810 15:20:58.505635 11966 ts_manager.cc:194] Registered new tserver with Master: 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915)
I20260810 15:20:58.505759 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.483908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:40297 (local address 127.4.64.126:41729)
0810 15:20:58.483956 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.483958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.484767 (+ 809us) server_negotiation.cc:407] Connection header received
0810 15:20:58.484800 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.484801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.484813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.484871 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.485852 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.486605 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.488698 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.488968 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.489004 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.495578 (+ 6574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.495581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.495582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.495593 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.501786 (+ 6193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.504577 (+ 2791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.504578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.504579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.504656 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.504921 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.504922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.504923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.505105 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.505207 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.505498 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.505598 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:20:58.506621 12188 ts_manager.cc:194] Registered new tserver with Master: 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915)
I20260810 15:20:58.506846 12557 heartbeater.cc:347] Connected to a master server at 127.4.64.126:41729
I20260810 15:20:58.506904 12557 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.506989 12557 heartbeater.cc:511] Master 127.4.64.126:41729 requested a full tablet report, sending...
I20260810 15:20:58.507244 11289 ts_manager.cc:194] Re-registered known tserver with Master: 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915)
I20260810 15:20:58.507688 11289 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:40297
I20260810 15:20:58.598716 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.595323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.4.64.124:40099)
0810 15:20:58.595402 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.595404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.595416 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:58.595450 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.595451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.595480 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.595571 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.595751 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.596594 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.597569 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.598267 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.598356 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.598444 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.598583 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.598633 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:20:58.598694 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.595188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58794)
0810 15:20:58.595273 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.595289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.595347 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.595597 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.595598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.595600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.595676 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.595677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.596709 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.596709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.597465 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.597467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.598434 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.598435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.598496 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.598500 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.598563 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.598604 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 15:20:58.599232 12199 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 15:20:58.599421 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b412a7829a2d4a5eb3c9c7b8d977cdf0. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.599535 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: This master's current role is: LEARNER
W20260810 15:20:58.602674 12563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:58.602963 12563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:58.603011 12563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:58.604440 12563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:58.604511 12563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:58.604564 12563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:20:58.606509 12563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:43981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=40469
--tserver_master_addrs=127.4.64.126:41729,127.4.64.125:34561,127.4.64.124:40099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:58.606815 12563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:58.607089 12563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:58.613128 12569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.613121 12570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:58.613121 12573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:58.613512 12563 server_base.cc:1034] running on GCE node
I20260810 15:20:58.613783 12563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:58.614063 12563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:58.615219 12563 hybrid_clock.cc:648] HybridClock initialized: now 1786375258615200 us; error 28 us; skew 500 ppm
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:58.618245 12563 init.cc:377] Logged in from keytab as oryx/127.4.64.67@KRBTEST.COM (short username oryx)
I20260810 15:20:58.618752 12563 webserver.cc:460] Webserver started at http://127.4.64.67:40469/ using document root <none> and password file <none>
I20260810 15:20:58.618984 12563 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:58.619045 12563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:58.620491 12563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:20:58.621138 12582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.621443 12563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:58.621543 12563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "d14f9b1454274be79c7893fe538ba840"
format_stamp: "Formatted at 2026-08-10 15:20:57 on dist-test-slave-9zdj"
server_key: "da32eda0665f66bf4da7cb4374b42163"
server_key_iv: "0bde924a8294d6f2a77391474d60480e"
server_key_version: "encryptionkey@0"
I20260810 15:20:58.621963 12563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:58.640219 12563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:58.642721 12563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:58.642886 12563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:58.643260 12563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:20:58.643676 12563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:20:58.643719 12563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.643745 12563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:20:58.643797 12563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:58.652324 12563 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43981
I20260810 15:20:58.652424 12719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43981 every 8 connection(s)
I20260810 15:20:58.652796 12563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.124@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.125@KRBTEST.COM
Aug 10 15:20:58 dist-test-slave-9zdj krb5kdc[11237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375258, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:20:58.662072 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 12563
I20260810 15:20:58.665691 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.653575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:36785)
0810 15:20:58.653772 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.653817 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.654378 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.654655 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.654660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.655135 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.655407 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.655413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.656393 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.656394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.657393 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.657396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.657973 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.658081 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.658509 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.658514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.661407 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.663596 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.663607 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.663610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.663997 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.664273 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.664274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.664275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.664440 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.664797 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.664805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.665004 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.665155 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.665369 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":57,"thread_start_us":132,"threads_started":1}
I20260810 15:20:58.665982 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.653905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.4.64.67:43607)
0810 15:20:58.654561 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.654571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.654649 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.654880 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.654881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.655129 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.655407 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.655413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.656393 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.656394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.657429 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.657431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.657929 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.658005 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.658681 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.658687 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.660360 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.664573 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.664574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.664575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.664874 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.665108 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.665109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.665109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.665184 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.665566 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.665567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.665647 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.665723 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.665842 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":65,"thread_start_us":78,"threads_started":1}
I20260810 15:20:58.666234 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.653581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:36785 (local address 127.4.64.126:41729)
0810 15:20:58.653658 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.653661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.653908 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:20:58.654481 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.654482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.654496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.654555 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.655517 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.656298 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657512 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.657834 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657883 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.661566 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.661568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.661569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.661578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.663483 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.664100 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.664102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.664102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.664172 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.664511 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.664512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.664512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.664685 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.664764 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.665959 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.666143 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:20:58.666316 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.653984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:43607 (local address 127.4.64.124:40099)
0810 15:20:58.654034 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.654036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.654623 (+ 587us) server_negotiation.cc:407] Connection header received
0810 15:20:58.654762 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.654763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.654775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.654805 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.655517 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.656289 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657514 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.657830 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657913 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.660486 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.660487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.660488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.660495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.664412 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.664965 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.664966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.664967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.665030 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.665292 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.665294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.665294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.665466 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.665539 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.666113 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.666224 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:20:58.666973 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.653575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.4.64.67:51493)
0810 15:20:58.654051 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.654063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.654394 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.655019 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.655020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.655183 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:58.655405 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.655410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.656185 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.656186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.657099 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.657101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.657638 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.657790 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.658400 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:58.658429 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:58.660873 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:58.664984 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.664985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.664985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.665263 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.665505 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:58.665506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:58.665507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:58.665575 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:58.666608 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:58.666609 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:58.666663 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.666743 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.666858 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":33,"thread_start_us":118,"threads_started":1}
I20260810 15:20:58.667335 12720 heartbeater.cc:347] Connected to a master server at 127.4.64.124:40099
I20260810 15:20:58.667449 12720 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.667598 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.654056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:51493 (local address 127.4.64.125:34561)
0810 15:20:58.654109 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.654111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.654123 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:58.654495 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.654496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.654510 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.654572 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.655518 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.656073 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657222 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.657513 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.657604 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.660973 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.660975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.660976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.660985 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.664484 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.665354 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.665355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.665356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.665425 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.665683 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.665684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.665685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.665849 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.665917 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.667409 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.667502 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:20:58.667649 12720 heartbeater.cc:511] Master 127.4.64.124:40099 requested a full tablet report, sending...
I20260810 15:20:58.667351 12721 heartbeater.cc:347] Connected to a master server at 127.4.64.126:41729
I20260810 15:20:58.667853 12721 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.667915 12721 heartbeater.cc:511] Master 127.4.64.126:41729 requested a full tablet report, sending...
I20260810 15:20:58.668190 12723 heartbeater.cc:347] Connected to a master server at 127.4.64.125:34561
I20260810 15:20:58.668226 12723 heartbeater.cc:464] Registering TS with master...
I20260810 15:20:58.668282 12723 heartbeater.cc:511] Master 127.4.64.125:34561 requested a full tablet report, sending...
I20260810 15:20:58.668937 11289 ts_manager.cc:194] Re-registered known tserver with Master: d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981)
I20260810 15:20:58.668951 12188 ts_manager.cc:194] Registered new tserver with Master: d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981)
I20260810 15:20:58.668951 11966 ts_manager.cc:194] Registered new tserver with Master: d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981)
I20260810 15:20:58.669390 11289 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:36785
I20260810 15:20:58.674029 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.663243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.4.64.126:41729)
0810 15:20:58.664715 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.664716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.664724 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:58.664742 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.664743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.664752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.665697 (+ 945us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.665739 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.666594 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.667750 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.668056 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.668141 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.670889 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.670891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.670891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.670898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.672667 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.673058 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.673059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.673060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.673140 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.673373 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.673374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.673374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.673525 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.673650 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.673805 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.673893 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1443,"thread_start_us":93,"threads_started":1}
I20260810 15:20:58.674430 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.663530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.4.64.125:34561)
0810 15:20:58.664523 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.664525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.664532 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:58.664555 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.664556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.664568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.664637 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.667135 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.667840 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.668895 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.669169 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.669229 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.670843 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.670844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.670845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.670852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.672799 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.673232 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.673233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.673234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.673317 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.673574 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.673575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.673575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.673751 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.673875 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.674230 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.674325 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":57,"threads_started":1}
I20260810 15:20:58.675804 11289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 15:20:58.677256 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.663919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.4.64.124:40099)
0810 15:20:58.664153 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.664155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.669686 (+ 5531us) server_negotiation.cc:407] Connection header received
0810 15:20:58.669717 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.669718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.669733 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.669860 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.670497 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.671257 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.672380 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.672761 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.672813 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.674355 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.674357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.674357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.674364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.676118 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.676454 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.676456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.676456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.676517 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.676721 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.676722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.676723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.676845 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.676951 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.677046 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.677138 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":57,"threads_started":1}
W20260810 15:20:58.677443 11289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:20:58.678370 11923 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: NON_VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: true } } }
I20260810 15:20:58.678474 11923 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.679674 12148 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: attempting to promote NON_VOTER ffb9e1aaf0ce46a681870b85b75b5fbe to VOTER
I20260810 15:20:58.679847 12148 consensus_queue.cc:237] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:20:58.680279 12199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 LEARNER]: Refusing update from remote peer b412a7829a2d4a5eb3c9c7b8d977cdf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 15:20:58.680279 11979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Refusing update from remote peer b412a7829a2d4a5eb3c9c7b8d977cdf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 15:20:58.680503 12149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 15:20:58.680651 12149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 15:20:58.681221 12149 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.681388 12199 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.681396 11979 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.682291 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.682380 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.682330 12238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.682410 12238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:58.682435 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.682484 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.682485 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.682533 12149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:20:58.682572 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:58.682580 12149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:58.687775 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.683768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.4.64.66:38915)
0810 15:20:58.684256 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.684262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.684272 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:58.684320 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.684322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.684339 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.684639 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.684702 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.685858 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.686735 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.687437 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.687518 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.687601 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.687658 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.687693 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":93,"threads_started":1}
I20260810 15:20:58.687765 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.683531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38915 (local address 127.0.0.1:36316)
0810 15:20:58.683960 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.683970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.684019 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.684448 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.684449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.684452 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.684493 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.684494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.685926 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.685927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.686644 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.686645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.687517 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.687518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.687580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.687585 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.687641 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.687687 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":102,"threads_started":1}
I20260810 15:20:58.688143 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.683428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45551 (local address 127.0.0.1:56046)
0810 15:20:58.683594 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.683607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.683651 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.684134 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.684134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.684136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.684193 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.684195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.685862 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.685863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.686931 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.686933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.687901 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.687902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.687952 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.687955 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.687983 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.688014 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260810 15:20:58.688333 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.683765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.4.64.65:45551)
0810 15:20:58.683947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.683952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.683962 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:58.683997 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.683999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.684012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.684130 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.684281 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.685767 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.687053 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.687807 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.688067 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.688197 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.688222 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.688262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260810 15:20:58.688345 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.683561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.4.64.67:43981)
0810 15:20:58.684152 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.684156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.684166 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:58.684207 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.684209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.684225 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.684302 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.684512 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.685334 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.686818 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.687617 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.687687 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.687740 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.688115 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.688162 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":94,"threads_started":1}
I20260810 15:20:58.688022 12748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.683461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43981 (local address 127.0.0.1:37038)
0810 15:20:58.683594 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.683607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.683651 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.684371 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.684371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.684373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.684430 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.684432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.685429 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.685430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.686599 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.686602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.687778 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.687779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.687846 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.687851 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.687902 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.687941 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260810 15:20:58.689023 12478 tablet_service.cc:1467] Processing CreateTablet for tablet 3f59999a3c274a00935895c889c25833 (DEFAULT_TABLE table=test-table [id=82a9c5c728da41f7af8bab214d4eb3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:58.689033 12318 tablet_service.cc:1467] Processing CreateTablet for tablet 3f59999a3c274a00935895c889c25833 (DEFAULT_TABLE table=test-table [id=82a9c5c728da41f7af8bab214d4eb3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:58.689179 12639 tablet_service.cc:1467] Processing CreateTablet for tablet 3f59999a3c274a00935895c889c25833 (DEFAULT_TABLE table=test-table [id=82a9c5c728da41f7af8bab214d4eb3b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:20:58.689421 12318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f59999a3c274a00935895c889c25833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:58.689426 12478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f59999a3c274a00935895c889c25833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:58.689483 12639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f59999a3c274a00935895c889c25833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:58.692708 12754 tablet_bootstrap.cc:492] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Bootstrap starting.
I20260810 15:20:58.693316 12757 tablet_bootstrap.cc:492] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Bootstrap starting.
I20260810 15:20:58.693384 12755 tablet_bootstrap.cc:492] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Bootstrap starting.
I20260810 15:20:58.693711 12754 tablet_bootstrap.cc:654] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:58.694031 12757 tablet_bootstrap.cc:654] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:58.694088 12754 log.cc:824] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:58.694154 12755 tablet_bootstrap.cc:654] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:58.694397 12757 log.cc:824] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:58.694520 12755 log.cc:824] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:58.695076 12754 tablet_bootstrap.cc:492] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: No bootstrap required, opened a new log
I20260810 15:20:58.695168 12754 ts_tablet_manager.cc:1397] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:20:58.695190 12757 tablet_bootstrap.cc:492] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: No bootstrap required, opened a new log
I20260810 15:20:58.695253 12757 ts_tablet_manager.cc:1397] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:20:58.695356 12755 tablet_bootstrap.cc:492] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: No bootstrap required, opened a new log
I20260810 15:20:58.695423 12755 ts_tablet_manager.cc:1397] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:20:58.697198 12757 raft_consensus.cc:348] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.697249 12754 raft_consensus.cc:348] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.697371 12757 raft_consensus.cc:374] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:58.697422 12757 raft_consensus.cc:729] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2ace53b1824f5ebb9e26a12f98d2a5, State: Initialized, Role: FOLLOWER
I20260810 15:20:58.697425 12754 raft_consensus.cc:374] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:58.697464 12754 raft_consensus.cc:729] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ee1bc0f4634e9db852e812838ff817, State: Initialized, Role: FOLLOWER
I20260810 15:20:58.697530 12755 raft_consensus.cc:348] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.697649 12754 consensus_queue.cc:260] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.697700 12755 raft_consensus.cc:374] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:58.697736 12755 raft_consensus.cc:729] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14f9b1454274be79c7893fe538ba840, State: Initialized, Role: FOLLOWER
I20260810 15:20:58.697547 12757 consensus_queue.cc:260] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.697870 12755 consensus_queue.cc:260] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.697947 12754 ts_tablet_manager.cc:1428] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:20:58.698096 12557 heartbeater.cc:503] Master 127.4.64.126:41729 was elected leader, sending a full tablet report...
I20260810 15:20:58.698166 12721 heartbeater.cc:503] Master 127.4.64.126:41729 was elected leader, sending a full tablet report...
I20260810 15:20:58.698103 12757 ts_tablet_manager.cc:1428] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 15:20:58.699230 12399 heartbeater.cc:503] Master 127.4.64.126:41729 was elected leader, sending a full tablet report...
I20260810 15:20:58.700469 12755 ts_tablet_manager.cc:1428] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260810 15:20:58.706646 12762 raft_consensus.cc:482] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:20:58.706801 12762 raft_consensus.cc:504] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.707299 12762 leader_election.cc:288] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551), d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981)
I20260810 15:20:58.711467 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.707570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:60703 (local address 127.4.64.65:45551)
0810 15:20:58.707621 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.707623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.707633 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:58.707696 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.707697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.707714 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.707776 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.707960 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.708850 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.710427 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.711065 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.711129 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.711140 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.711348 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.711397 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:20:58.711467 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.707587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:43363 (local address 127.4.64.67:43981)
0810 15:20:58.707713 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.707715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.707737 (+ 22us) server_negotiation.cc:407] Connection header received
0810 15:20:58.707775 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.707777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.707788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.707836 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:20:58.708014 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.708844 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.710427 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.711025 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.711116 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.711129 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.711363 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.711404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260810 15:20:58.711463 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.707470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43981 (local address 127.4.64.66:43363)
0810 15:20:58.707559 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.707575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.707629 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.707881 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.707881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.707883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.707935 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.707937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.708953 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.708954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.710326 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.710330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.711143 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.711144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.711291 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.711294 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.711348 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.711379 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":72}
I20260810 15:20:58.711481 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.707478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45551 (local address 127.4.64.66:60703)
0810 15:20:58.707588 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:58.707600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:58.707646 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:58.707789 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:58.707790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:58.707792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:20:58.707881 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.707883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.708953 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.708954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.710326 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:58.710330 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.711143 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.711144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:58.711241 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.711245 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:20:58.711315 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:20:58.711366 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":128}
I20260810 15:20:58.711860 12663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f59999a3c274a00935895c889c25833" candidate_uuid: "30ee1bc0f4634e9db852e812838ff817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d14f9b1454274be79c7893fe538ba840" is_pre_election: true
I20260810 15:20:58.711941 12344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f59999a3c274a00935895c889c25833" candidate_uuid: "30ee1bc0f4634e9db852e812838ff817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" is_pre_election: true
I20260810 15:20:58.712057 12663 raft_consensus.cc:2454] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ee1bc0f4634e9db852e812838ff817 in term 0.
I20260810 15:20:58.712100 12344 raft_consensus.cc:2454] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30ee1bc0f4634e9db852e812838ff817 in term 0.
I20260810 15:20:58.712641 12428 leader_election.cc:302] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ee1bc0f4634e9db852e812838ff817, d14f9b1454274be79c7893fe538ba840; no voters:
I20260810 15:20:58.712777 12762 raft_consensus.cc:2781] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:20:58.712834 12762 raft_consensus.cc:482] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:20:58.712862 12762 raft_consensus.cc:3037] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:58.713730 12762 raft_consensus.cc:504] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.713876 12762 leader_election.cc:288] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [CANDIDATE]: Term 1 election: Requested vote from peers bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551), d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981)
I20260810 15:20:58.714066 12344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f59999a3c274a00935895c889c25833" candidate_uuid: "30ee1bc0f4634e9db852e812838ff817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5"
I20260810 15:20:58.714154 12344 raft_consensus.cc:3037] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:58.714835 12344 raft_consensus.cc:2454] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ee1bc0f4634e9db852e812838ff817 in term 1.
I20260810 15:20:58.715034 12426 leader_election.cc:302] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ee1bc0f4634e9db852e812838ff817, bd2ace53b1824f5ebb9e26a12f98d2a5; no voters:
I20260810 15:20:58.715194 12762 raft_consensus.cc:2781] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:58.715114 12663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f59999a3c274a00935895c889c25833" candidate_uuid: "30ee1bc0f4634e9db852e812838ff817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d14f9b1454274be79c7893fe538ba840"
I20260810 15:20:58.715308 12663 raft_consensus.cc:3037] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:58.715405 12762 raft_consensus.cc:686] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 1 LEADER]: Becoming Leader. State: Replica: 30ee1bc0f4634e9db852e812838ff817, State: Running, Role: LEADER
I20260810 15:20:58.715516 12762 consensus_queue.cc:237] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } }
I20260810 15:20:58.716884 12663 raft_consensus.cc:2454] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30ee1bc0f4634e9db852e812838ff817 in term 1.
I20260810 15:20:58.717415 11288 catalog_manager.cc:5495] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66). New cstate: current_term: 1 leader_uuid: "30ee1bc0f4634e9db852e812838ff817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } health_report { overall_health: UNKNOWN } } }
I20260810 15:20:58.726857 12752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.723891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.4.64.66:38915)
0810 15:20:58.723938 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.723940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.723951 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:58.724035 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.724036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.724064 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.724148 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:58.724327 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.724915 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.725979 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.726199 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.726305 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.726500 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:58.726613 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:58.726715 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.726738 (+ 23us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.726764 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:58.729077 12344 raft_consensus.cc:1261] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 1 FOLLOWER]: Refusing update from remote peer 30ee1bc0f4634e9db852e812838ff817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:58.729076 12663 raft_consensus.cc:1261] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Refusing update from remote peer 30ee1bc0f4634e9db852e812838ff817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:58.729456 12762 consensus_queue.cc:1035] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:58.729612 12765 consensus_queue.cc:1035] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:58.732393 12769 mvcc.cc:205] Tried to move back new op lower bound from 7316993059750965248 to 7316993059699376128. Current Snapshot: MvccSnapshot[applied={T|T < 7316993059750965248}]
I20260810 15:20:58.734011 12771 mvcc.cc:205] Tried to move back new op lower bound from 7316993059750965248 to 7316993059699376128. Current Snapshot: MvccSnapshot[applied={T|T < 7316993059750965248}]
I20260810 15:20:58.736644 11288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:58.736750 11288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:20:58.738392 11288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b412a7829a2d4a5eb3c9c7b8d977cdf0: Sending DeleteTablet for 3 replicas of tablet 3f59999a3c274a00935895c889c25833
I20260810 15:20:58.738734 12318 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f59999a3c274a00935895c889c25833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:58 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:56046
I20260810 15:20:58.738845 12639 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f59999a3c274a00935895c889c25833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:58 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37038
I20260810 15:20:58.739009 12782 tablet_replica.cc:330] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: stopping tablet replica
I20260810 15:20:58.739149 12782 raft_consensus.cc:2229] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:58.739204 12783 tablet_replica.cc:330] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: stopping tablet replica
I20260810 15:20:58.739187 12478 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f59999a3c274a00935895c889c25833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:20:58 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:36316
I20260810 15:20:58.739322 12783 raft_consensus.cc:2229] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:20:58.739338 12782 raft_consensus.cc:2258] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:58.739435 12783 raft_consensus.cc:2258] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:58.739952 12782 ts_tablet_manager.cc:1905] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:58.739943 12783 ts_tablet_manager.cc:1905] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:58.740597 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 11253
I20260810 15:20:58.745020 12783 ts_tablet_manager.cc:1918] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:58.745151 12783 log.cc:1196] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/3f59999a3c274a00935895c889c25833
I20260810 15:20:58.745065 12782 ts_tablet_manager.cc:1918] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:58.745385 12782 log.cc:1196] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/3f59999a3c274a00935895c889c25833
I20260810 15:20:58.745471 12783 ts_tablet_manager.cc:1939] T 3f59999a3c274a00935895c889c25833 P d14f9b1454274be79c7893fe538ba840: Deleting consensus metadata
I20260810 15:20:58.745857 12782 ts_tablet_manager.cc:1939] T 3f59999a3c274a00935895c889c25833 P bd2ace53b1824f5ebb9e26a12f98d2a5: Deleting consensus metadata
I20260810 15:20:58.746394 12784 tablet_replica.cc:330] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: stopping tablet replica
I20260810 15:20:58.746598 12784 raft_consensus.cc:2229] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:20:58.746829 12784 raft_consensus.cc:2258] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:20:58.747735 12784 ts_tablet_manager.cc:1905] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:20:58.750033 12784 ts_tablet_manager.cc:1918] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:20:58.750195 12784 log.cc:1196] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/3f59999a3c274a00935895c889c25833
I20260810 15:20:58.750604 12784 ts_tablet_manager.cc:1939] T 3f59999a3c274a00935895c889c25833 P 30ee1bc0f4634e9db852e812838ff817: Deleting consensus metadata
W20260810 15:20:58.752189 12427 connection.cc:535] server connection from 127.0.0.1:36316 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 15:20:58.752252 12584 connection.cc:535] server connection from 127.0.0.1:37038 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 15:20:58.752482 12269 connection.cc:535] server connection from 127.0.0.1:56046 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:20:58.756187 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.740080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.4.64.124:40099)
0810 15:20:58.740122 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.740124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.747169 (+ 7045us) server_negotiation.cc:407] Connection header received
0810 15:20:58.747211 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.747212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.747230 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.747296 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.748144 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.748940 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.750101 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.750447 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.750492 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.752230 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.752231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.752232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.752240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.754214 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.754615 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.754617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.754617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.754701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.754983 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.754984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.754985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.755153 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.755249 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.755942 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.756058 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:58.764026 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.754845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.4.64.125:34561)
0810 15:20:58.754894 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.754896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.754905 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:20:58.754931 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.754932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.754949 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.755013 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.755638 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.756438 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.757699 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.758065 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.758146 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.760033 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.760034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.760035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.760043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.762330 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.762937 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.762939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.762939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.763010 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.763310 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.763311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.763312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.763477 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.763581 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.763830 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.763927 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:20:58.763935 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:58.755387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.4.64.124:40099)
0810 15:20:58.755443 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:58.755445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:58.755831 (+ 386us) server_negotiation.cc:407] Connection header received
0810 15:20:58.755937 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:58.755938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:58.755950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:58.755982 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:58.756722 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.757459 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.758564 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:58.758867 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:58.758940 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:58.760492 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:58.760493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:58.760494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:58.760502 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:58.762506 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.763037 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.763039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.763039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.763100 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:58.763341 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:58.763342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:58.763343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:58.763470 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:58.763583 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:58.763718 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:20:58.763808 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:20:59.734431 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:59.734010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:47801)
0810 15:20:59.734264 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:59.734365 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":73,"threads_started":1}
W20260810 15:20:59.734691 12557 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:41729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:20:59.735224 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:59.734924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:49139)
0810 15:20:59.735103 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:59.735178 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":56,"threads_started":1}
I20260810 15:20:59.735224 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:59.734942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:58203)
0810 15:20:59.735100 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:59.735177 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
W20260810 15:20:59.735420 12721 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:41729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
W20260810 15:20:59.735519 12399 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:41729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.239423 12798 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b412a7829a2d4a5eb3c9c7b8d977cdf0)
I20260810 15:21:00.239478 12799 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b412a7829a2d4a5eb3c9c7b8d977cdf0)
I20260810 15:21:00.239529 12798 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:00.239674 12799 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:00.239773 12798 leader_election.cc:288] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), a0009246e17642e4aeec69a80efed348 (127.4.64.125:34561)
I20260810 15:21:00.240017 12799 leader_election.cc:288] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124:40099)
I20260810 15:21:00.240202 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.239929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39660)
0810 15:21:00.240092 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.240166 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":56,"threads_started":1}
W20260810 15:21:00.240433 12171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.240568 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.240109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39672)
0810 15:21:00.240470 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.240530 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":76,"threads_started":1}
W20260810 15:21:00.240670 11952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.240900 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.240791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39684)
0810 15:21:00.240844 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.240872 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 15:21:00.241003 12171 leader_election.cc:334] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.241130 12803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.241041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39690)
0810 15:21:00.241080 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.241106 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 15:21:00.241242 11952 leader_election.cc:334] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.244557 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.240451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.4.64.124:40099)
0810 15:21:00.240862 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:00.240865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:00.240871 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:21:00.240892 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:00.240893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:00.240902 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:00.241133 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:00.241161 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.241951 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.243535 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.244126 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.244194 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:00.244207 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:00.244410 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 15:21:00.244455 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":42,"threads_started":1}
I20260810 15:21:00.244557 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.240018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.4.64.125:34561)
0810 15:21:00.240116 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:00.240119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:00.240517 (+ 398us) server_negotiation.cc:407] Connection header received
0810 15:21:00.240552 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:00.240553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:00.240568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:00.240645 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:00.241054 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.241951 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.243535 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.244112 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.244194 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:00.244207 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:00.244405 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 15:21:00.244455 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":44,"threads_started":1}
I20260810 15:21:00.244575 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.240342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:40099 (local address 127.0.0.1:58828)
0810 15:21:00.240715 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.240725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:00.240769 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:00.241005 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:00.241006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:00.241009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:00.241061 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:00.241063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.242045 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.242046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:00.243428 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:00.243431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.244230 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.244231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:00.244306 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:00.244311 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:21:00.244386 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:21:00.244434 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":57,"threads_started":1}
I20260810 15:21:00.244573 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.239962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:34561 (local address 127.0.0.1:51856)
0810 15:21:00.240145 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.240159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:00.240330 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:00.240717 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:00.240719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:00.240722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:00.240844 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:00.240847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.242046 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.242047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:00.243428 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:00.243431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:00.244230 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:00.244231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:00.244317 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:00.244321 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:00.244386 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:21:00.244435 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
I20260810 15:21:00.244925 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0009246e17642e4aeec69a80efed348" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" is_pre_election: true
I20260810 15:21:00.245211 12199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0009246e17642e4aeec69a80efed348 in term 1.
I20260810 15:21:00.245321 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a0009246e17642e4aeec69a80efed348" is_pre_election: true
I20260810 15:21:00.245390 11952 leader_election.cc:302] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0009246e17642e4aeec69a80efed348, ffb9e1aaf0ce46a681870b85b75b5fbe; no voters: b412a7829a2d4a5eb3c9c7b8d977cdf0
I20260810 15:21:00.245399 11979 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffb9e1aaf0ce46a681870b85b75b5fbe in term 1.
I20260810 15:21:00.245505 12799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 15:21:00.245553 12799 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b412a7829a2d4a5eb3c9c7b8d977cdf0)
I20260810 15:21:00.245579 12799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:21:00.245673 12169 leader_election.cc:302] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0009246e17642e4aeec69a80efed348, ffb9e1aaf0ce46a681870b85b75b5fbe; no voters: b412a7829a2d4a5eb3c9c7b8d977cdf0
I20260810 15:21:00.245787 12798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 15:21:00.245823 12798 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Starting leader election (detected failure of leader b412a7829a2d4a5eb3c9c7b8d977cdf0)
I20260810 15:21:00.245841 12798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:21:00.246363 12799 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:00.246496 12799 leader_election.cc:288] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 2 election: Requested vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124:40099)
I20260810 15:21:00.246624 12798 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:00.246727 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0009246e17642e4aeec69a80efed348" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
I20260810 15:21:00.246791 12798 leader_election.cc:288] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 2 election: Requested vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), a0009246e17642e4aeec69a80efed348 (127.4.64.125:34561)
I20260810 15:21:00.246824 12199 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a0009246e17642e4aeec69a80efed348 in current term 2: Already voted for candidate ffb9e1aaf0ce46a681870b85b75b5fbe in this term.
I20260810 15:21:00.246752 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.246602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39706)
0810 15:21:00.246682 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.246711 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 15:21:00.246951 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.246850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39718)
0810 15:21:00.246889 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.246922 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 15:21:00.246939 11979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a0009246e17642e4aeec69a80efed348"
I20260810 15:21:00.247005 11979 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ffb9e1aaf0ce46a681870b85b75b5fbe in current term 2: Already voted for candidate a0009246e17642e4aeec69a80efed348 in this term.
I20260810 15:21:00.247270 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.247165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39730)
0810 15:21:00.247218 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.247246 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 15:21:00.247347 12171 leader_election.cc:334] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.247399 12171 leader_election.cc:302] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffb9e1aaf0ce46a681870b85b75b5fbe; no voters: a0009246e17642e4aeec69a80efed348, b412a7829a2d4a5eb3c9c7b8d977cdf0
I20260810 15:21:00.247493 12798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 15:21:00.247536 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.247268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:39746)
0810 15:21:00.247317 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.247512 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:21:00.247607 11952 leader_election.cc:334] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:00.247689 11952 leader_election.cc:302] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a0009246e17642e4aeec69a80efed348; no voters: b412a7829a2d4a5eb3c9c7b8d977cdf0, ffb9e1aaf0ce46a681870b85b75b5fbe
I20260810 15:21:00.247767 12799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 15:21:00.735695 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.735240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:56653)
0810 15:21:00.735461 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.735504 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260810 15:21:00.736042 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.735792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:55155)
0810 15:21:00.735959 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.735994 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260810 15:21:00.736171 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:00.735874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:55223)
0810 15:21:00.736085 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:00.736123 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":66,"threads_started":1}
I20260810 15:21:01.736693 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:01.736458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:33801)
0810 15:21:01.736594 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:01.736640 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260810 15:21:01.736693 12815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:01.736459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:48617)
0810 15:21:01.736607 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:01.736638 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260810 15:21:01.736933 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:01.736537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:60205)
0810 15:21:01.736714 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:01.736756 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":46,"threads_started":1}
I20260810 15:21:02.295771 12817 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:02.295884 12817 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:02.296092 12817 leader_election.cc:288] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124:40099)
I20260810 15:21:02.296490 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.296286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:40060)
0810 15:21:02.296397 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.296442 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260810 15:21:02.296595 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0009246e17642e4aeec69a80efed348" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" is_pre_election: true
I20260810 15:21:02.296703 12199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0009246e17642e4aeec69a80efed348 in term 2.
I20260810 15:21:02.296998 11952 leader_election.cc:302] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0009246e17642e4aeec69a80efed348, ffb9e1aaf0ce46a681870b85b75b5fbe; no voters:
I20260810 15:21:02.297084 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.296948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:40064)
0810 15:21:02.297004 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.297036 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:21:02.297109 12817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 15:21:02.297150 12817 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:02.297175 12817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 2 FOLLOWER]: Advancing to term 3
W20260810 15:21:02.297202 11952 leader_election.cc:334] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:02.298038 12817 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:02.298172 12817 leader_election.cc:288] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 3 election: Requested vote from peers b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729), ffb9e1aaf0ce46a681870b85b75b5fbe (127.4.64.124:40099)
I20260810 15:21:02.298381 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.298261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:40076)
0810 15:21:02.298319 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.298352 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:21:02.298446 12199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0009246e17642e4aeec69a80efed348" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe"
I20260810 15:21:02.298538 12199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:21:02.298678 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.298606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:40082)
0810 15:21:02.298633 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.298658 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260810 15:21:02.298776 11952 leader_election.cc:334] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
I20260810 15:21:02.299335 12199 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0009246e17642e4aeec69a80efed348 in term 3.
I20260810 15:21:02.299531 11952 leader_election.cc:302] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a0009246e17642e4aeec69a80efed348, ffb9e1aaf0ce46a681870b85b75b5fbe; no voters: b412a7829a2d4a5eb3c9c7b8d977cdf0
I20260810 15:21:02.299614 12817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:21:02.299746 12817 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [term 3 LEADER]: Becoming Leader. State: Replica: a0009246e17642e4aeec69a80efed348, State: Running, Role: LEADER
I20260810 15:21:02.299835 12817 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } }
I20260810 15:21:02.300241 12823 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0009246e17642e4aeec69a80efed348. Latest consensus state: current_term: 3 leader_uuid: "a0009246e17642e4aeec69a80efed348" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:21:02.300324 12823 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:02.300544 12827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:02.301041 12827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:02.301249 12827 catalog_manager.cc:1259] Loaded cluster ID: c8e6b649a7064ed891dd7928bb9254c0
I20260810 15:21:02.301303 12827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:21:02.301496 12827 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:02.301589 12827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Loaded TSK: 0
I20260810 15:21:02.301764 12827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:21:02.347944 11966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:21:02.349849 11966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:02.350960 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.350839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:40108)
0810 15:21:02.350891 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.350932 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:21:02.350970 12199 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [term 3 FOLLOWER]: Refusing update from remote peer a0009246e17642e4aeec69a80efed348: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 15:21:02.351375 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.351244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.0.0.1:40110)
0810 15:21:02.351320 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.351350 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 15:21:02.351377 12823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 15:21:02.351620 11952 consensus_peers.cc:487] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 -> Peer b412a7829a2d4a5eb3c9c7b8d977cdf0 (127.4.64.126:41729): Couldn't send request to peer b412a7829a2d4a5eb3c9c7b8d977cdf0. Status: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 15:21:02.352275 12830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0009246e17642e4aeec69a80efed348. Latest consensus state: current_term: 3 leader_uuid: "a0009246e17642e4aeec69a80efed348" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:21:02.352411 12830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:21:02.352748 12829 mvcc.cc:205] Tried to move back new op lower bound from 7316993074587348992 to 7316993074380828672. Current Snapshot: MvccSnapshot[applied={T|T < 7316993074587348992}]
I20260810 15:21:02.352850 12817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a0009246e17642e4aeec69a80efed348" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:21:02.352965 12817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:02.352972 12832 mvcc.cc:205] Tried to move back new op lower bound from 7316993074587348992 to 7316993074380828672. Current Snapshot: MvccSnapshot[applied={T|T < 7316993074587348992}]
I20260810 15:21:02.353228 12830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a0009246e17642e4aeec69a80efed348" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b412a7829a2d4a5eb3c9c7b8d977cdf0" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41729 } } peers { permanent_uuid: "a0009246e17642e4aeec69a80efed348" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 34561 } attrs { promote: false } } peers { permanent_uuid: "ffb9e1aaf0ce46a681870b85b75b5fbe" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 40099 } attrs { promote: false } } }
I20260810 15:21:02.353319 12830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffb9e1aaf0ce46a681870b85b75b5fbe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:21:02.358385 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.354856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.4.64.66:38915)
0810 15:21:02.354976 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:02.354978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:02.354986 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:02.355020 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:02.355021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:02.355034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:02.355113 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:02.355306 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.356098 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.357435 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.358000 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.358079 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.358094 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:02.358241 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:21:02.358299 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260810 15:21:02.358368 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.354890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45551 (local address 127.0.0.1:56058)
0810 15:21:02.355074 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.355084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:02.355122 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:02.355501 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:02.355502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:02.355503 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:02.355554 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.355556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.356271 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.356272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.357132 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.357135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.358115 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.358116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.358192 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.358198 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:21:02.358228 (+ 30us) client_negotiation.cc:240] Negotiation successful
0810 15:21:02.358272 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260810 15:21:02.358444 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.354784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38915 (local address 127.0.0.1:36336)
0810 15:21:02.354859 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.354877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:02.354934 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:02.355126 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:02.355128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:02.355130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:02.355191 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.355194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.356204 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.356205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.357333 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.357336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.358090 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.358092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.358158 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.358162 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:02.358232 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:21:02.358312 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":25}
I20260810 15:21:02.358769 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.355088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.4.64.65:45551)
0810 15:21:02.355217 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:02.355219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:02.355227 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:02.355254 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:02.355255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:02.355266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:02.355502 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:02.355607 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.356196 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.357222 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.358006 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.358605 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.358622 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:02.358642 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 15:21:02.358695 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260810 15:21:02.358850 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.354893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.4.64.67:43981)
0810 15:21:02.355227 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:02.355230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:02.355377 (+ 147us) server_negotiation.cc:407] Connection header received
0810 15:21:02.355433 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:02.355434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:02.355446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:02.355515 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:02.355768 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.356522 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.357805 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.358394 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.358478 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.358496 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:02.358715 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 15:21:02.358761 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":61,"threads_started":1}
I20260810 15:21:02.358978 12318 tablet_service.cc:1467] Processing CreateTablet for tablet 538c4554fc124f21bb3216091b8ac76b (DEFAULT_TABLE table=test-table [id=2d3313f1a3d046479b6ec60c688dfe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:02.359002 12478 tablet_service.cc:1467] Processing CreateTablet for tablet 538c4554fc124f21bb3216091b8ac76b (DEFAULT_TABLE table=test-table [id=2d3313f1a3d046479b6ec60c688dfe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:02.359045 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.354790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43981 (local address 127.0.0.1:37044)
0810 15:21:02.355299 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.355310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:02.355373 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:02.355622 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:02.355623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:02.355624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:02.355674 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.355675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.356603 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.356604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.357724 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.357727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.358534 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.358535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.358614 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.358618 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:02.358931 (+ 313us) client_negotiation.cc:240] Negotiation successful
0810 15:21:02.358979 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":390,"thread_start_us":74,"threads_started":1}
I20260810 15:21:02.359115 12318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538c4554fc124f21bb3216091b8ac76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:02.359153 12478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538c4554fc124f21bb3216091b8ac76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:02.359351 12639 tablet_service.cc:1467] Processing CreateTablet for tablet 538c4554fc124f21bb3216091b8ac76b (DEFAULT_TABLE table=test-table [id=2d3313f1a3d046479b6ec60c688dfe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:02.359458 12639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 538c4554fc124f21bb3216091b8ac76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:02.360476 12843 tablet_bootstrap.cc:492] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Bootstrap starting.
I20260810 15:21:02.360992 12845 tablet_bootstrap.cc:492] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Bootstrap starting.
I20260810 15:21:02.361101 12843 tablet_bootstrap.cc:654] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:02.361135 12844 tablet_bootstrap.cc:492] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5: Bootstrap starting.
I20260810 15:21:02.361385 12845 tablet_bootstrap.cc:654] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:02.361562 12844 tablet_bootstrap.cc:654] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:02.361822 12843 tablet_bootstrap.cc:492] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: No bootstrap required, opened a new log
I20260810 15:21:02.361904 12843 ts_tablet_manager.cc:1397] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:02.362031 12843 raft_consensus.cc:348] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.362102 12843 raft_consensus.cc:374] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:02.362118 12843 raft_consensus.cc:729] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ee1bc0f4634e9db852e812838ff817, State: Initialized, Role: FOLLOWER
I20260810 15:21:02.362185 12845 tablet_bootstrap.cc:492] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: No bootstrap required, opened a new log
I20260810 15:21:02.362195 12844 tablet_bootstrap.cc:492] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5: No bootstrap required, opened a new log
I20260810 15:21:02.362175 12843 consensus_queue.cc:260] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.362234 12845 ts_tablet_manager.cc:1397] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:02.362233 12844 ts_tablet_manager.cc:1397] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:02.362339 12843 ts_tablet_manager.cc:1428] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:02.362349 12844 raft_consensus.cc:348] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.362412 12844 raft_consensus.cc:374] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:02.362435 12844 raft_consensus.cc:729] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2ace53b1824f5ebb9e26a12f98d2a5, State: Initialized, Role: FOLLOWER
I20260810 15:21:02.362349 12845 raft_consensus.cc:348] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.362478 12844 consensus_queue.cc:260] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.362651 12844 ts_tablet_manager.cc:1428] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 15:21:02.363193 12851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.362579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:57207)
0810 15:21:02.363111 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.363155 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":81,"threads_started":1}
I20260810 15:21:02.363423 12558 heartbeater.cc:503] Master 127.4.64.125:34561 was elected leader, sending a full tablet report...
I20260810 15:21:02.363565 12400 heartbeater.cc:503] Master 127.4.64.125:34561 was elected leader, sending a full tablet report...
I20260810 15:21:02.362465 12845 raft_consensus.cc:374] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:02.363940 12845 raft_consensus.cc:729] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14f9b1454274be79c7893fe538ba840, State: Initialized, Role: FOLLOWER
I20260810 15:21:02.363993 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.363648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:42695)
0810 15:21:02.363912 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.363952 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260810 15:21:02.364111 12845 consensus_queue.cc:260] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.364234 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.363377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:50449)
0810 15:21:02.364150 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.364190 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":82,"threads_started":1}
I20260810 15:21:02.364368 12845 ts_tablet_manager.cc:1428] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:21:02.364691 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.364459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:44257)
0810 15:21:02.364581 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.364636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260810 15:21:02.364636 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.364456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:44089)
0810 15:21:02.364583 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.364609 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260810 15:21:02.364883 12723 heartbeater.cc:503] Master 127.4.64.125:34561 was elected leader, sending a full tablet report...
I20260810 15:21:02.365191 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.365059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:37271)
0810 15:21:02.365138 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.365163 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 15:21:02.406807 12724 tablet.cc:2369] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:21:02.414459 12856 raft_consensus.cc:482] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:02.414574 12856 raft_consensus.cc:504] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.414964 12856 leader_election.cc:288] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915), bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551)
I20260810 15:21:02.419342 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.415217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:41041 (local address 127.4.64.66:38915)
0810 15:21:02.415264 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:02.415271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:02.415365 (+ 94us) server_negotiation.cc:407] Connection header received
0810 15:21:02.415461 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:02.415462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:02.415479 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:02.415555 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:02.415785 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.416608 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.418301 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.418872 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.418953 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.418966 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:02.419208 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 15:21:02.419269 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:21:02.419355 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.415129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38915 (local address 127.4.64.67:41041)
0810 15:21:02.415281 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.415294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:02.415365 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:02.415572 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:02.415573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:02.415575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:02.415708 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.415710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.416726 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.416727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.418211 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.418215 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.418966 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.418967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.419094 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.419099 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:21:02.419191 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:21:02.419247 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":76}
I20260810 15:21:02.419427 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.415434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:44147 (local address 127.4.64.65:45551)
0810 15:21:02.415495 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:02.415496 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:02.415566 (+ 70us) server_negotiation.cc:407] Connection header received
0810 15:21:02.415622 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:02.415623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:02.415633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:02.415688 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:02.415835 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.416589 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.418422 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.418968 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.419032 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.419047 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:02.419283 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 15:21:02.419325 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:21:02.419637 12501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538c4554fc124f21bb3216091b8ac76b" candidate_uuid: "d14f9b1454274be79c7893fe538ba840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ee1bc0f4634e9db852e812838ff817" is_pre_election: true
I20260810 15:21:02.419369 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.415313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45551 (local address 127.4.64.67:44147)
0810 15:21:02.415484 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.415493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:02.415569 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:02.415706 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:02.415707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:02.415708 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:02.415759 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.415761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.416726 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.416727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.418333 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:02.418335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.419084 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.419085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:02.419155 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.419159 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:02.419257 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 15:21:02.419288 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":164,"thread_start_us":123,"threads_started":1}
I20260810 15:21:02.419803 12501 raft_consensus.cc:2454] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14f9b1454274be79c7893fe538ba840 in term 0.
I20260810 15:21:02.420058 12583 leader_election.cc:302] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ee1bc0f4634e9db852e812838ff817, d14f9b1454274be79c7893fe538ba840; no voters:
I20260810 15:21:02.420106 12344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538c4554fc124f21bb3216091b8ac76b" candidate_uuid: "d14f9b1454274be79c7893fe538ba840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" is_pre_election: true
I20260810 15:21:02.420202 12344 raft_consensus.cc:2454] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d14f9b1454274be79c7893fe538ba840 in term 0.
I20260810 15:21:02.420354 12856 raft_consensus.cc:2781] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:02.420406 12856 raft_consensus.cc:482] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:02.420430 12856 raft_consensus.cc:3037] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:02.421181 12856 raft_consensus.cc:504] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.421329 12856 leader_election.cc:288] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [CANDIDATE]: Term 1 election: Requested vote from peers 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915), bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551)
I20260810 15:21:02.421491 12501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538c4554fc124f21bb3216091b8ac76b" candidate_uuid: "d14f9b1454274be79c7893fe538ba840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ee1bc0f4634e9db852e812838ff817"
I20260810 15:21:02.421514 12344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538c4554fc124f21bb3216091b8ac76b" candidate_uuid: "d14f9b1454274be79c7893fe538ba840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5"
I20260810 15:21:02.421584 12501 raft_consensus.cc:3037] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:02.421584 12344 raft_consensus.cc:3037] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:02.422477 12344 raft_consensus.cc:2454] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14f9b1454274be79c7893fe538ba840 in term 1.
I20260810 15:21:02.422514 12501 raft_consensus.cc:2454] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d14f9b1454274be79c7893fe538ba840 in term 1.
I20260810 15:21:02.422721 12583 leader_election.cc:302] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd2ace53b1824f5ebb9e26a12f98d2a5, d14f9b1454274be79c7893fe538ba840; no voters:
I20260810 15:21:02.422827 12856 raft_consensus.cc:2781] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:02.423002 12856 raft_consensus.cc:686] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 1 LEADER]: Becoming Leader. State: Replica: d14f9b1454274be79c7893fe538ba840, State: Running, Role: LEADER
I20260810 15:21:02.423090 12856 consensus_queue.cc:237] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } }
I20260810 15:21:02.423447 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.423251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:59511)
0810 15:21:02.423369 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.423406 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 15:21:02.423995 11965 catalog_manager.cc:5495] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 reported cstate change: term changed from 0 to 1, leader changed from <none> to d14f9b1454274be79c7893fe538ba840 (127.4.64.67). New cstate: current_term: 1 leader_uuid: "d14f9b1454274be79c7893fe538ba840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d14f9b1454274be79c7893fe538ba840" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 } health_report { overall_health: UNKNOWN } } }
I20260810 15:21:02.433416 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.430736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.4.64.67:43981)
0810 15:21:02.430787 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:02.430789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:02.430812 (+ 23us) server_negotiation.cc:407] Connection header received
0810 15:21:02.430879 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:02.430879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:02.430899 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:02.430956 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:02.431065 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.431570 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.432481 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:02.432764 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:02.432823 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:02.432948 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:02.433061 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:02.433115 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:02.433270 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:21:02.433293 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:21:02.434747 12344 raft_consensus.cc:1261] T 538c4554fc124f21bb3216091b8ac76b P bd2ace53b1824f5ebb9e26a12f98d2a5 [term 1 FOLLOWER]: Refusing update from remote peer d14f9b1454274be79c7893fe538ba840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:02.434876 12501 raft_consensus.cc:1261] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Refusing update from remote peer d14f9b1454274be79c7893fe538ba840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:02.435053 12863 consensus_queue.cc:1035] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2ace53b1824f5ebb9e26a12f98d2a5" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:02.435190 12856 consensus_queue.cc:1035] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ee1bc0f4634e9db852e812838ff817" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:02.435953 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.435835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:60725)
0810 15:21:02.435882 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.435917 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:21:02.436360 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.436166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:54745)
0810 15:21:02.436293 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.436327 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 15:21:02.437932 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.437509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.67:36237)
0810 15:21:02.437868 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.437902 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20260810 15:21:02.438306 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.438212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.65:35563)
0810 15:21:02.438254 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.438276 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 15:21:02.439275 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.439112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:41729 (local address 127.4.64.66:39999)
0810 15:21:02.439210 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.439238 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:41729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 15:21:02.439633 11964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:02.439739 11964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:02.441073 11964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0009246e17642e4aeec69a80efed348: Sending DeleteTablet for 3 replicas of tablet 538c4554fc124f21bb3216091b8ac76b
I20260810 15:21:02.441497 12478 tablet_service.cc:1514] Processing DeleteTablet for tablet 538c4554fc124f21bb3216091b8ac76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:02 UTC) from {username='oryx', principal='oryx/127.4.64.125@KRBTEST.COM'} at 127.0.0.1:36336
I20260810 15:21:02.441550 12318 tablet_service.cc:1514] Processing DeleteTablet for tablet 538c4554fc124f21bb3216091b8ac76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:02 UTC) from {username='oryx', principal='oryx/127.4.64.125@KRBTEST.COM'} at 127.0.0.1:56058
I20260810 15:21:02.441751 12880 tablet_replica.cc:330] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: stopping tablet replica
I20260810 15:21:02.441810 12880 raft_consensus.cc:2229] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:02.441865 12880 raft_consensus.cc:2258] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:02.441893 12639 tablet_service.cc:1514] Processing DeleteTablet for tablet 538c4554fc124f21bb3216091b8ac76b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:02 UTC) from {username='oryx', principal='oryx/127.4.64.125@KRBTEST.COM'} at 127.0.0.1:37044
I20260810 15:21:02.442217 12880 ts_tablet_manager.cc:1905] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:02.442315 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 12142
I20260810 15:21:02.443902 12882 tablet_replica.cc:330] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: stopping tablet replica
I20260810 15:21:02.443988 12882 raft_consensus.cc:2229] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:21:02.444090 12882 raft_consensus.cc:2258] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:02.444312 12880 ts_tablet_manager.cc:1918] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:02.444386 12880 log.cc:1196] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/538c4554fc124f21bb3216091b8ac76b
I20260810 15:21:02.444721 12880 ts_tablet_manager.cc:1939] T 538c4554fc124f21bb3216091b8ac76b P 30ee1bc0f4634e9db852e812838ff817: Deleting consensus metadata
I20260810 15:21:02.445109 11948 catalog_manager.cc:4841] TS 30ee1bc0f4634e9db852e812838ff817 (127.4.64.66:38915): tablet 538c4554fc124f21bb3216091b8ac76b (table test-table [id=2d3313f1a3d046479b6ec60c688dfe57]) successfully deleted
I20260810 15:21:02.445248 12882 ts_tablet_manager.cc:1905] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:02.446831 12882 ts_tablet_manager.cc:1918] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:02.446907 12882 log.cc:1196] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/538c4554fc124f21bb3216091b8ac76b
I20260810 15:21:02.447253 12882 ts_tablet_manager.cc:1939] T 538c4554fc124f21bb3216091b8ac76b P d14f9b1454274be79c7893fe538ba840: Deleting consensus metadata
I20260810 15:21:02.448808 11951 catalog_manager.cc:4841] TS d14f9b1454274be79c7893fe538ba840 (127.4.64.67:43981): tablet 538c4554fc124f21bb3216091b8ac76b (table test-table [id=2d3313f1a3d046479b6ec60c688dfe57]) successfully deleted
I20260810 15:21:02.455389 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:02.455254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:45551 (local address 127.0.0.1:56070)
0810 15:21:02.455308 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:02.455338 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:45551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:21:02.455560 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 12408
W20260810 15:21:02.455576 11948 catalog_manager.cc:4584] TS bd2ace53b1824f5ebb9e26a12f98d2a5 (127.4.64.65:45551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 538c4554fc124f21bb3216091b8ac76b: Network error: Client connection negotiation failed: client connection to 127.4.64.65:45551: connect: Connection refused (error 111)
I20260810 15:21:02.466147 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 12563
I20260810 15:21:02.472579 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 11813
I20260810 15:21:02.482403 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 12028
2026-08-10T15:21:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6892 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:21:02 dist-test-slave-9zdj krb5kdc[12887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:21:02 dist-test-slave-9zdj krb5kdc[12887](info): set up 2 sockets
Aug 10 15:21:02 dist-test-slave-9zdj krb5kdc[12887](info): commencing operation
krb5kdc: starting...
W20260810 15:21:03.699235 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:21:03 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab.
Entry for principal oryx/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab.
I20260810 15:21:03.837489 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34659
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab
--principal=oryx/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:03.930547 12903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:03.930796 12903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:03.930835 12903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:03.932238 12903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:03.932319 12903 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 15:21:03.932340 12903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:03.932361 12903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:03.932391 12903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:03.932415 12903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:03.934094 12903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34659
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.126.keytab
--principal=oryx/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:03.934402 12903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:03.934638 12903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:03.940218 12908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:03.940260 12909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:03.940235 12911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:03.940717 12903 server_base.cc:1034] running on GCE node
I20260810 15:21:03.941035 12903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:03.941383 12903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:03.942577 12903 hybrid_clock.cc:648] HybridClock initialized: now 1786375263942537 us; error 55 us; skew 500 ppm
Aug 10 15:21:03 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375263, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:03.955691 12903 init.cc:377] Logged in from keytab as oryx/127.4.64.126@KRBTEST.COM (short username oryx)
I20260810 15:21:03.956269 12903 webserver.cc:460] Webserver started at http://127.4.64.126:37859/ using document root <none> and password file <none>
I20260810 15:21:03.956554 12903 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:03.956624 12903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:03.956794 12903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:03.958098 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "529105e32f98436390a1c239efd92f6e"
format_stamp: "Formatted at 2026-08-10 15:21:03 on dist-test-slave-9zdj"
server_key: "632c552827b04b7b575b8ca41291d9da"
server_key_iv: "a1e211c31ae32ed2d7d903246c1eef02"
server_key_version: "encryptionkey@0"
I20260810 15:21:03.958482 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "529105e32f98436390a1c239efd92f6e"
format_stamp: "Formatted at 2026-08-10 15:21:03 on dist-test-slave-9zdj"
server_key: "632c552827b04b7b575b8ca41291d9da"
server_key_iv: "a1e211c31ae32ed2d7d903246c1eef02"
server_key_version: "encryptionkey@0"
I20260810 15:21:03.959996 12903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:03.960805 12918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:03.961035 12903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:03.961146 12903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "529105e32f98436390a1c239efd92f6e"
format_stamp: "Formatted at 2026-08-10 15:21:03 on dist-test-slave-9zdj"
server_key: "632c552827b04b7b575b8ca41291d9da"
server_key_iv: "a1e211c31ae32ed2d7d903246c1eef02"
server_key_version: "encryptionkey@0"
I20260810 15:21:03.961221 12903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:03.996814 12903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:03.999197 12903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:03.999388 12903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:04.005347 12903 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34659
I20260810 15:21:04.005417 12996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34659 every 8 connection(s)
I20260810 15:21:04.005831 12903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:21:04.006494 12997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.007396 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 12903
I20260810 15:21:04.007534 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:21:04.007752 4353 external_mini_cluster.cc:1428] Setting key 49067f020d9a61517d71a68e38bbf3f0
I20260810 15:21:04.009620 12997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Bootstrap starting.
I20260810 15:21:04.010309 12997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.010607 12997 log.cc:824] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:04.011325 12997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: No bootstrap required, opened a new log
I20260810 15:21:04.013227 12997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529105e32f98436390a1c239efd92f6e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34659 } }
I20260810 15:21:04.013417 12997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.013455 12997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529105e32f98436390a1c239efd92f6e, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.013586 12997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529105e32f98436390a1c239efd92f6e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34659 } }
I20260810 15:21:04.013715 12997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:21:04.013773 12997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:21:04.013825 12997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.014362 12997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529105e32f98436390a1c239efd92f6e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34659 } }
I20260810 15:21:04.014483 12997 leader_election.cc:302] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 529105e32f98436390a1c239efd92f6e; no voters:
I20260810 15:21:04.014667 12997 leader_election.cc:288] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:21:04.014732 13002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:04.014905 13002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [term 1 LEADER]: Becoming Leader. State: Replica: 529105e32f98436390a1c239efd92f6e, State: Running, Role: LEADER
I20260810 15:21:04.014959 12997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:21:04.014991 13002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529105e32f98436390a1c239efd92f6e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34659 } }
I20260810 15:21:04.015271 13003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "529105e32f98436390a1c239efd92f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529105e32f98436390a1c239efd92f6e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34659 } } }
I20260810 15:21:04.015338 13003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [sys.catalog]: This master's current role is: LEADER
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:04.015496 13004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529105e32f98436390a1c239efd92f6e. Latest consensus state: current_term: 1 leader_uuid: "529105e32f98436390a1c239efd92f6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529105e32f98436390a1c239efd92f6e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34659 } } }
I20260810 15:21:04.015552 13004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:04.015538 13007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:04.016052 13007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:04.017800 13007 catalog_manager.cc:1347] Generated new cluster ID: 9b1b909cbbaa4aa7be890afc4537e528
I20260810 15:21:04.017850 13007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:21:04.019989 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.008423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.4.64.126:34659)
0810 15:21:04.008633 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.008643 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.008680 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:21:04.009328 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.009337 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.009411 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.009569 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.010604 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.011767 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.012526 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.012976 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.013100 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.016043 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.016047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.016047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.016060 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.018309 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.018730 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.018733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.018733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.018837 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.019058 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.019058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.019059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.019353 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.019495 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.019645 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.019780 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260810 15:21:04.023573 13007 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:21:04.024307 13007 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:04.035660 13007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Generated new TSK 0
I20260810 15:21:04.035925 13007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab.
Entry for principal oryx/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab.
I20260810 15:21:04.072428 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34659
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:04.160698 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:04.160912 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:04.160935 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:04.162151 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:04.162194 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:04.162214 13027 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:21:04.162235 13027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:21:04.163829 13027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:34659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:04.164076 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:04.164356 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:04.170715 13038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:04.170658 13040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:04.170691 13037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:04.170912 13027 server_base.cc:1034] running on GCE node
I20260810 15:21:04.171185 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:04.171412 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:04.172564 13027 hybrid_clock.cc:648] HybridClock initialized: now 1786375264172551 us; error 35 us; skew 500 ppm
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:04.175514 13027 init.cc:377] Logged in from keytab as oryx/127.4.64.65@KRBTEST.COM (short username oryx)
I20260810 15:21:04.176005 13027 webserver.cc:460] Webserver started at http://127.4.64.65:35759/ using document root <none> and password file <none>
I20260810 15:21:04.176270 13027 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:04.176317 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:04.176483 13027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:04.177668 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "052dbc5f756b477b84b75f9776655720"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "fdf476da22113af852e208ade23942cd"
server_key_iv: "8ce22b356907e8694cbe21fa3bbea8ed"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.178054 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "052dbc5f756b477b84b75f9776655720"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "fdf476da22113af852e208ade23942cd"
server_key_iv: "8ce22b356907e8694cbe21fa3bbea8ed"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.179530 13027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:04.180470 13047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.180660 13027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:04.180753 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "052dbc5f756b477b84b75f9776655720"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "fdf476da22113af852e208ade23942cd"
server_key_iv: "8ce22b356907e8694cbe21fa3bbea8ed"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.180823 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:04.190344 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:04.192770 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:04.192895 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:04.193500 13027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:04.193573 13027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.193608 13027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:04.193657 13027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:04.201126 13027 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:46509
I20260810 15:21:04.201231 13183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:46509 every 8 connection(s)
I20260810 15:21:04.201552 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:21:04.204255 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.193973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:50355 (local address 127.4.64.126:34659)
0810 15:21:04.194041 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.194044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.194278 (+ 234us) server_negotiation.cc:407] Connection header received
0810 15:21:04.194820 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.194822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.194836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.194902 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.195777 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.196639 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.197832 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.198050 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.198125 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.201155 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.201156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.201156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.201165 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.202807 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.203195 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.203196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.203196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.203251 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.203555 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.203556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.203557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.203709 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.203785 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.204016 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.204113 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:21:04.204270 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.193957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.4.64.65:50355)
0810 15:21:04.194132 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.194173 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.194710 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.194978 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.194982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.195349 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.195650 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.195657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.196756 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.196758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.197730 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.197733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.198159 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.198160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.198302 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.198868 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.198879 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.201043 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.202886 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.202892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.202894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.203127 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.203366 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.203366 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.203366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.203471 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.203780 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.203785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.203902 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.203983 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.204111 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260810 15:21:04.204780 13184 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34659
I20260810 15:21:04.204883 13184 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:04.205063 13184 heartbeater.cc:511] Master 127.4.64.126:34659 requested a full tablet report, sending...
I20260810 15:21:04.205730 12940 ts_manager.cc:194] Registered new tserver with Master: 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509)
I20260810 15:21:04.206547 12940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:50355
I20260810 15:21:04.208781 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 13027
I20260810 15:21:04.208899 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:21:04.209167 4353 external_mini_cluster.cc:1428] Setting key d7de5cf0083b10d278c82287c81368e7
I20260810 15:21:04.212224 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.206092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:51461 (local address 127.4.64.126:34659)
0810 15:21:04.206145 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.206148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.206281 (+ 133us) server_negotiation.cc:407] Connection header received
0810 15:21:04.206323 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.206324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.206334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.206398 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.206804 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.207356 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.208119 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.208304 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.208363 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.209497 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.209499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.209499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.209507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.211074 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.211405 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.211406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.211407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.211471 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.211648 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.211649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.211649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.211800 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.211868 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.212050 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.212141 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:21:04.212360 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.206032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.4.64.65:51461)
0810 15:21:04.206214 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.206224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.206279 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.206402 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.206403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.206565 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.206666 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.206667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.207469 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.207470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.208047 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.208049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.208374 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.208375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.208429 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.208854 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.208861 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.209385 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.211162 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.211163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.211164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.211339 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.211537 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.211537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.211538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.211590 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.211861 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.211862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.211899 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.212007 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.212139 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab.
Entry for principal oryx/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab.
I20260810 15:21:04.241528 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34659
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:04.327661 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:04.327872 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:04.327898 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:04.328979 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:04.329013 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:04.329034 13195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:21:04.329051 13195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:21:04.330508 13195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:34659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:04.330700 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:04.330884 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:04.336292 13200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:04.336361 13195 server_base.cc:1034] running on GCE node
W20260810 15:21:04.336303 13201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:04.336519 13203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:04.336800 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:04.337045 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:04.338200 13195 hybrid_clock.cc:648] HybridClock initialized: now 1786375264338199 us; error 37 us; skew 500 ppm
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:04.340729 13195 init.cc:377] Logged in from keytab as oryx/127.4.64.66@KRBTEST.COM (short username oryx)
I20260810 15:21:04.341199 13195 webserver.cc:460] Webserver started at http://127.4.64.66:38243/ using document root <none> and password file <none>
I20260810 15:21:04.341394 13195 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:04.341436 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:04.341528 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:04.342638 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "d999b9cc80924aa5b59681d5573fe67f"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "7a18ae60ae98fcae6a218661a95ff403"
server_key_iv: "1397eb67754c7ed9b225c1f5f959a62b"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.342957 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "d999b9cc80924aa5b59681d5573fe67f"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "7a18ae60ae98fcae6a218661a95ff403"
server_key_iv: "1397eb67754c7ed9b225c1f5f959a62b"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.344265 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:21:04.344982 13212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.345197 13195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:04.345296 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "d999b9cc80924aa5b59681d5573fe67f"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "7a18ae60ae98fcae6a218661a95ff403"
server_key_iv: "1397eb67754c7ed9b225c1f5f959a62b"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.345374 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:04.368115 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:04.370756 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:04.370922 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:04.371625 13195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:04.371680 13195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.371702 13195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:04.371737 13195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.378953 13195 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:34303
I20260810 15:21:04.379015 13350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:34303 every 8 connection(s)
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:04.379390 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:21:04.382640 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.372106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:57069 (local address 127.4.64.126:34659)
0810 15:21:04.372176 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.372178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.372406 (+ 228us) server_negotiation.cc:407] Connection header received
0810 15:21:04.372904 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.372906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.372920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.373007 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.373915 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.374789 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.376433 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.376808 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.376893 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.379781 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.379783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.379783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.379791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.381278 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.381731 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.381732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.381733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.381788 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.382014 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.382015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.382015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.382127 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.382176 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.382507 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.382577 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:21:04.382856 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.372079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.4.64.66:57069)
0810 15:21:04.372253 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.372296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.372806 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.373100 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.373103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.373458 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.373800 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.373808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.374891 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.374892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.376332 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.376336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.376904 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.376905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.377055 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.377663 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.377671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.379665 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.381356 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.381363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.381365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.381591 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.381845 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.381845 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.381846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.381954 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.382259 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.382266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.382389 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.382471 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.382602 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260810 15:21:04.383399 13351 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34659
I20260810 15:21:04.383488 13351 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:04.383702 13351 heartbeater.cc:511] Master 127.4.64.126:34659 requested a full tablet report, sending...
I20260810 15:21:04.384119 12941 ts_manager.cc:194] Registered new tserver with Master: d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303)
I20260810 15:21:04.384637 12941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:57069
I20260810 15:21:04.388080 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 13195
I20260810 15:21:04.388212 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:21:04.388459 4353 external_mini_cluster.cc:1428] Setting key 5032844a84b2d684400bac4b8375de29
I20260810 15:21:04.390236 13357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.384494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.4.64.66:43595)
0810 15:21:04.384629 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.384641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.384738 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.384883 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.384883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.385113 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.385194 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.385197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.385925 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.385926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.386463 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.386465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.386768 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.386769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.386815 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.387183 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.387187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.387459 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.389165 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.389166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.389166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.389330 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.389522 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.389523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.389523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.389574 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.389955 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.389956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.390001 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.390072 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.390158 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":7,"thread_start_us":67,"threads_started":1}
I20260810 15:21:04.390331 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.384550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:43595 (local address 127.4.64.126:34659)
0810 15:21:04.384595 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.384596 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.384721 (+ 125us) server_negotiation.cc:407] Connection header received
0810 15:21:04.384812 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.384813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.384823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.384873 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.385286 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.385841 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.386531 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.386709 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.386756 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.387528 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.387529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.387529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.387535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.389091 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.389397 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.389398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.389398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.389459 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.389646 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.389647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.389647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.389865 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.389942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.390115 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.390201 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab.
Entry for principal oryx/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab.
I20260810 15:21:04.417987 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34659
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:04.504272 13360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:04.504480 13360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:04.504505 13360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:04.505812 13360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:04.505863 13360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:04.505884 13360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:21:04.505903 13360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:21:04.507473 13360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.64.126:34659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:04.507735 13360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:04.507980 13360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:04.513379 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:04.513396 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:04.513408 13360 server_base.cc:1034] running on GCE node
W20260810 15:21:04.513379 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:04.513878 13360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:04.514144 13360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:04.515300 13360 hybrid_clock.cc:648] HybridClock initialized: now 1786375264515286 us; error 29 us; skew 500 ppm
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:04.518064 13360 init.cc:377] Logged in from keytab as oryx/127.4.64.67@KRBTEST.COM (short username oryx)
I20260810 15:21:04.518586 13360 webserver.cc:460] Webserver started at http://127.4.64.67:43029/ using document root <none> and password file <none>
I20260810 15:21:04.518842 13360 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:04.518913 13360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:04.519061 13360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:04.520180 13360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "5e9d1f26e8554b868c2d771738935e07"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "a02d0de934283896b227020af67e3d86"
server_key_iv: "ccd18e61dfae5c1cf1a5946e14c9d613"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.520565 13360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "5e9d1f26e8554b868c2d771738935e07"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "a02d0de934283896b227020af67e3d86"
server_key_iv: "ccd18e61dfae5c1cf1a5946e14c9d613"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.522050 13360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:04.523079 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.523313 13360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:04.523422 13360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "5e9d1f26e8554b868c2d771738935e07"
format_stamp: "Formatted at 2026-08-10 15:21:04 on dist-test-slave-9zdj"
server_key: "a02d0de934283896b227020af67e3d86"
server_key_iv: "ccd18e61dfae5c1cf1a5946e14c9d613"
server_key_version: "encryptionkey@0"
I20260810 15:21:04.523505 13360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:04.552868 13360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:04.555425 13360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:04.555601 13360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:04.556143 13360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:04.556226 13360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.556298 13360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:04.556326 13360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:04.563958 13360 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:37829
I20260810 15:21:04.564069 13516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:37829 every 8 connection(s)
I20260810 15:21:04.564448 13360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:21:04.566587 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.556867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:52511 (local address 127.4.64.126:34659)
0810 15:21:04.556922 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.556924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.557074 (+ 150us) server_negotiation.cc:407] Connection header received
0810 15:21:04.557768 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.557770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.557785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.557842 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.558492 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.559022 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.560133 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.560338 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.560423 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.563306 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.563308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.563308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.563318 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.565134 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.565572 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.565573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.565574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.565684 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.565930 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.565931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.565931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.566048 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.566116 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.566451 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.566523 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:21:04.566838 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.556744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.4.64.67:52511)
0810 15:21:04.556932 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.556985 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.557623 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.557917 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.557922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.558184 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.558389 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.558393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.559128 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.559129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.560043 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.560046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.560445 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.560446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.560581 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.561146 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.561154 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.563193 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.565226 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.565233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.565235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.565506 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.565769 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.565770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.565770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.565875 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.566129 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.566135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.566317 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.566425 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.566562 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260810 15:21:04.567343 13517 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34659
I20260810 15:21:04.567430 13517 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:04.567631 13517 heartbeater.cc:511] Master 127.4.64.126:34659 requested a full tablet report, sending...
I20260810 15:21:04.568048 12940 ts_manager.cc:194] Registered new tserver with Master: 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:04.568557 12940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:52511
I20260810 15:21:04.574208 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.568515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:33279 (local address 127.4.64.126:34659)
0810 15:21:04.568559 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.568561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.568685 (+ 124us) server_negotiation.cc:407] Connection header received
0810 15:21:04.568735 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.568736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.568746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.568803 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.569262 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.569822 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.570639 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.570846 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.570905 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.571948 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.571949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.571950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.571955 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.573176 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.573558 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.573558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.573558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.573658 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.573813 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.573814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.573814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.573919 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.573971 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.574085 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.574142 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:21:04.574332 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.568450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.4.64.67:33279)
0810 15:21:04.568581 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.568592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.568677 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.568810 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.568811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.569050 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.569137 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.569140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.569913 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.569914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.570555 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.570558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.570917 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.570917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.570971 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.571552 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.571563 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.571870 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.573313 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.573314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.573314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.573495 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.573709 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.573710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.573710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.573762 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.573975 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.573976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.574016 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.574067 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.574230 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260810 15:21:04.574635 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 13360
I20260810 15:21:04.574749 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:21:04.574945 4353 external_mini_cluster.cc:1428] Setting key 8a0727c31e0212bc980d2820dc5417ac
I20260810 15:21:04.575766 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:21:04.599942 13000 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:04.594297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.4.64.126:34659)
0810 15:21:04.594357 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.594359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.594392 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:21:04.594419 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.594419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.594429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.594474 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.595069 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.595620 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.596454 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.596640 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.596692 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.599817 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37896: BlockingRecv error: recv got EOF from 127.0.0.1:37896 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:04.608837 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.600930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.4.64.126:34659)
0810 15:21:04.601050 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.601053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.601151 (+ 98us) server_negotiation.cc:407] Connection header received
0810 15:21:04.601230 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.601231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.601241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.601294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.601871 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.602366 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.603064 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.603266 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.603337 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.606089 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.606090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.606091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.606098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.607622 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.608009 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.608010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.608011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.608073 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.608278 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.608278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.608279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.608400 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.608507 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.608642 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.608727 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 15:21:04.611050 12940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:21:04.613000 12940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:04.620458 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.616030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.4.64.67:37829)
0810 15:21:04.616330 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.616336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.616348 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:21:04.616397 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.616400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.616416 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.616499 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.616698 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.617916 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.619292 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.620020 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.620122 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.620199 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.620320 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.620361 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260810 15:21:04.620442 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.615906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37829 (local address 127.0.0.1:59598)
0810 15:21:04.616045 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.616078 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.616153 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.616509 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.616512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.616515 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.616594 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.616596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.618000 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.618001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.619203 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.619205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.620121 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.620122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.620206 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.620221 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.620288 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.620348 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260810 15:21:04.620740 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.616364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.4.64.65:46509)
0810 15:21:04.616522 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.616526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.616714 (+ 188us) server_negotiation.cc:407] Connection header received
0810 15:21:04.616757 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.616760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.616773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.616865 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.617075 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.618127 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.619490 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.620194 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.620257 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.620326 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.620419 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.620459 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260810 15:21:04.620980 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.616578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.4.64.66:34303)
0810 15:21:04.617113 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.617117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.617124 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:21:04.617155 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.617156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.617167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.617248 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.617370 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.618293 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.619533 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.620027 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.620117 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.620197 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.620838 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.620877 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":93,"threads_started":1}
I20260810 15:21:04.620563 13545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.616141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46509 (local address 127.0.0.1:52340)
0810 15:21:04.616595 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.616606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.616656 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.616935 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.616936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.616938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.616998 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.616999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.618207 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.618208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.619388 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.619390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.620279 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.620280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.620352 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.620356 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.620407 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.620448 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":78,"threads_started":1}
I20260810 15:21:04.620924 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.616337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:34303 (local address 127.0.0.1:50818)
0810 15:21:04.616843 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.616853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.616905 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.617245 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.617246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.617248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.617313 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.617314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.618375 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.618376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.619447 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.619449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.620680 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.620681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.620746 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.620750 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.620821 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.620861 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":72,"threads_started":1}
I20260810 15:21:04.621742 13437 tablet_service.cc:1467] Processing CreateTablet for tablet 0c302ba96e624bf091b1ad363e361c75 (DEFAULT_TABLE table=test-table [id=6ffd1a9daff04dddb3c15d60a76e6762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:04.621819 13269 tablet_service.cc:1467] Processing CreateTablet for tablet 0c302ba96e624bf091b1ad363e361c75 (DEFAULT_TABLE table=test-table [id=6ffd1a9daff04dddb3c15d60a76e6762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:04.621809 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 0c302ba96e624bf091b1ad363e361c75 (DEFAULT_TABLE table=test-table [id=6ffd1a9daff04dddb3c15d60a76e6762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:04.622184 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c302ba96e624bf091b1ad363e361c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.622180 13437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c302ba96e624bf091b1ad363e361c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.622289 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c302ba96e624bf091b1ad363e361c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.625041 13549 tablet_bootstrap.cc:492] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Bootstrap starting.
I20260810 15:21:04.625798 13549 tablet_bootstrap.cc:654] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.626173 13549 log.cc:824] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:04.626183 13550 tablet_bootstrap.cc:492] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Bootstrap starting.
I20260810 15:21:04.626353 13551 tablet_bootstrap.cc:492] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Bootstrap starting.
I20260810 15:21:04.626917 13550 tablet_bootstrap.cc:654] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.626983 13551 tablet_bootstrap.cc:654] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.627101 13549 tablet_bootstrap.cc:492] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: No bootstrap required, opened a new log
I20260810 15:21:04.627172 13549 ts_tablet_manager.cc:1397] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:21:04.627332 13550 log.cc:824] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:04.627408 13551 log.cc:824] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:04.628082 13550 tablet_bootstrap.cc:492] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: No bootstrap required, opened a new log
I20260810 15:21:04.628157 13550 ts_tablet_manager.cc:1397] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:21:04.629288 13549 raft_consensus.cc:348] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.629468 13549 raft_consensus.cc:374] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.629522 13549 raft_consensus.cc:729] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d1f26e8554b868c2d771738935e07, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.629683 13549 consensus_queue.cc:260] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.629984 13549 ts_tablet_manager.cc:1428] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 15:21:04.630107 13517 heartbeater.cc:503] Master 127.4.64.126:34659 was elected leader, sending a full tablet report...
I20260810 15:21:04.630256 13550 raft_consensus.cc:348] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.630410 13550 raft_consensus.cc:374] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.630430 13551 tablet_bootstrap.cc:492] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: No bootstrap required, opened a new log
I20260810 15:21:04.630455 13550 raft_consensus.cc:729] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.630497 13551 ts_tablet_manager.cc:1397] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:21:04.630568 13550 consensus_queue.cc:260] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.630841 13550 ts_tablet_manager.cc:1428] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:21:04.630878 13351 heartbeater.cc:503] Master 127.4.64.126:34659 was elected leader, sending a full tablet report...
I20260810 15:21:04.632304 13551 raft_consensus.cc:348] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.632436 13551 raft_consensus.cc:374] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.632458 13551 raft_consensus.cc:729] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.632530 13551 consensus_queue.cc:260] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.632725 13551 ts_tablet_manager.cc:1428] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:21:04.632815 13184 heartbeater.cc:503] Master 127.4.64.126:34659 was elected leader, sending a full tablet report...
I20260810 15:21:04.682117 13558 raft_consensus.cc:482] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:04.682242 13558 raft_consensus.cc:504] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.682557 13558 leader_election.cc:288] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829), 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509)
I20260810 15:21:04.685781 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.682813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37829 (local address 127.4.64.66:38451)
0810 15:21:04.682877 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.682891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.682942 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.683178 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.683180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.683181 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.683252 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.683254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.683991 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.683992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.684898 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.684901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.685460 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.685461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.685556 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.685560 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.685654 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.685698 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:21:04.685781 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.682894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:38451 (local address 127.4.64.67:37829)
0810 15:21:04.682940 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.682942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.682956 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:21:04.683088 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.683089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.683101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.683168 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.683337 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.683910 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.684982 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.685367 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.685457 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.685470 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.685657 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.685698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:21:04.686249 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c302ba96e624bf091b1ad363e361c75" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07" is_pre_election: true
I20260810 15:21:04.686345 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.682953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46509 (local address 127.4.64.66:51459)
0810 15:21:04.683181 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.683191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.683266 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.683412 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.683412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.683414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.683495 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.683496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.684330 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.684331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.685353 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.685356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.686141 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.686142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.686200 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.686204 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.686250 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.686280 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":158,"threads_started":1}
I20260810 15:21:04.686398 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.683039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:51459 (local address 127.4.64.65:46509)
0810 15:21:04.683076 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.683078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.683261 (+ 183us) server_negotiation.cc:407] Connection header received
0810 15:21:04.683329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.683330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.683343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.683408 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.683636 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.684242 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.685453 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.686054 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.686145 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.686156 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.686277 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.686324 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:21:04.686447 13465 raft_consensus.cc:2454] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 0.
I20260810 15:21:04.686765 13216 leader_election.cc:302] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9d1f26e8554b868c2d771738935e07, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:04.686899 13558 raft_consensus.cc:2781] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:04.686967 13558 raft_consensus.cc:482] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:04.687031 13558 raft_consensus.cc:3037] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.687134 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c302ba96e624bf091b1ad363e361c75" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052dbc5f756b477b84b75f9776655720" is_pre_election: true
I20260810 15:21:04.687304 13130 raft_consensus.cc:2454] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 0.
I20260810 15:21:04.688150 13558 raft_consensus.cc:504] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.688283 13558 leader_election.cc:288] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 election: Requested vote from peers 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829), 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509)
I20260810 15:21:04.688431 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c302ba96e624bf091b1ad363e361c75" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07"
I20260810 15:21:04.688457 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c302ba96e624bf091b1ad363e361c75" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052dbc5f756b477b84b75f9776655720"
I20260810 15:21:04.688531 13130 raft_consensus.cc:3037] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.688537 13465 raft_consensus.cc:3037] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.689391 13130 raft_consensus.cc:2454] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 1.
I20260810 15:21:04.689513 13465 raft_consensus.cc:2454] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 1.
I20260810 15:21:04.689575 13216 leader_election.cc:302] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:04.689703 13558 raft_consensus.cc:2781] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:04.689915 13558 raft_consensus.cc:686] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 1 LEADER]: Becoming Leader. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Running, Role: LEADER
I20260810 15:21:04.690037 13558 consensus_queue.cc:237] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } }
I20260810 15:21:04.691205 12941 catalog_manager.cc:5495] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f reported cstate change: term changed from 0 to 1, leader changed from <none> to d999b9cc80924aa5b59681d5573fe67f (127.4.64.66). New cstate: current_term: 1 leader_uuid: "d999b9cc80924aa5b59681d5573fe67f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } health_report { overall_health: HEALTHY } } }
I20260810 15:21:04.699326 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.696921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.4.64.66:34303)
0810 15:21:04.696991 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.696993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.697000 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:21:04.697039 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.697040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.697057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.697132 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:04.697302 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.697835 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.698583 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.698769 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.698839 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.698995 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:04.699098 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:04.699189 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.699210 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.699240 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 15:21:04.701481 13465 raft_consensus.cc:1261] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Refusing update from remote peer d999b9cc80924aa5b59681d5573fe67f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:04.701493 13130 raft_consensus.cc:1261] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Refusing update from remote peer d999b9cc80924aa5b59681d5573fe67f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:04.701879 13558 consensus_queue.cc:1035] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [LEADER]: Connected to new peer: Peer: permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:04.702034 13561 consensus_queue.cc:1035] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:21:04.702713 13185 tablet.cc:2369] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:21:04.704882 13567 mvcc.cc:205] Tried to move back new op lower bound from 7316993084213669888 to 7316993084171276288. Current Snapshot: MvccSnapshot[applied={T|T < 7316993084213669888}]
I20260810 15:21:04.706245 13565 mvcc.cc:205] Tried to move back new op lower bound from 7316993084213669888 to 7316993084171276288. Current Snapshot: MvccSnapshot[applied={T|T < 7316993084213669888 or (T in {7316993084213669888})}]
I20260810 15:21:04.706259 13566 mvcc.cc:205] Tried to move back new op lower bound from 7316993084213669888 to 7316993084171276288. Current Snapshot: MvccSnapshot[applied={T|T < 7316993084213669888}]
I20260810 15:21:04.707265 12940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:04.707369 12940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:04.708213 12940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Sending DeleteTablet for 3 replicas of tablet 0c302ba96e624bf091b1ad363e361c75
I20260810 15:21:04.708604 13437 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c302ba96e624bf091b1ad363e361c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:04 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:59598
I20260810 15:21:04.708917 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c302ba96e624bf091b1ad363e361c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:04 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:50818
I20260810 15:21:04.708977 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c302ba96e624bf091b1ad363e361c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:04 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:52340
I20260810 15:21:04.709007 13575 tablet_replica.cc:330] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: stopping tablet replica
I20260810 15:21:04.709117 13575 raft_consensus.cc:2229] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:04.709146 12940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:21:04.709317 12940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:04.709753 13576 tablet_replica.cc:330] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: stopping tablet replica
I20260810 15:21:04.709754 13575 raft_consensus.cc:2258] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:04.709860 13576 raft_consensus.cc:2229] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:21:04.709959 13576 raft_consensus.cc:2258] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:04.710307 13575 ts_tablet_manager.cc:1905] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:04.710848 13437 tablet_service.cc:1467] Processing CreateTablet for tablet b237f0eb46d044b29d4f1a17465ef2d8 (DEFAULT_TABLE table=test-table [id=843a25cb84ba457893b42e1c5a2c79ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:04.710902 13269 tablet_service.cc:1467] Processing CreateTablet for tablet b237f0eb46d044b29d4f1a17465ef2d8 (DEFAULT_TABLE table=test-table [id=843a25cb84ba457893b42e1c5a2c79ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:04.710975 13437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b237f0eb46d044b29d4f1a17465ef2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.711031 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b237f0eb46d044b29d4f1a17465ef2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.712126 13575 ts_tablet_manager.cc:1918] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:04.712210 13575 log.cc:1196] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/0c302ba96e624bf091b1ad363e361c75
I20260810 15:21:04.712357 13549 tablet_bootstrap.cc:492] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Bootstrap starting.
I20260810 15:21:04.712471 13550 tablet_bootstrap.cc:492] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Bootstrap starting.
I20260810 15:21:04.712666 13103 tablet_service.cc:1467] Processing CreateTablet for tablet b237f0eb46d044b29d4f1a17465ef2d8 (DEFAULT_TABLE table=test-table [id=843a25cb84ba457893b42e1c5a2c79ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:04.712795 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b237f0eb46d044b29d4f1a17465ef2d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.712841 13549 tablet_bootstrap.cc:654] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.712894 13550 tablet_bootstrap.cc:654] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.713335 13575 ts_tablet_manager.cc:1939] T 0c302ba96e624bf091b1ad363e361c75 P 5e9d1f26e8554b868c2d771738935e07: Deleting consensus metadata
I20260810 15:21:04.714032 13550 tablet_bootstrap.cc:492] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: No bootstrap required, opened a new log
I20260810 15:21:04.714087 13550 ts_tablet_manager.cc:1397] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:21:04.714234 13549 tablet_bootstrap.cc:492] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: No bootstrap required, opened a new log
I20260810 15:21:04.714277 13549 ts_tablet_manager.cc:1397] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260810 15:21:04.714306 13551 tablet_bootstrap.cc:492] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Bootstrap starting.
I20260810 15:21:04.714262 13550 raft_consensus.cc:348] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.714354 13550 raft_consensus.cc:374] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.714382 13550 raft_consensus.cc:729] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.714432 13550 consensus_queue.cc:260] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.714429 13549 raft_consensus.cc:348] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.714495 13549 raft_consensus.cc:374] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.714519 13550 ts_tablet_manager.cc:1428] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.714541 13549 raft_consensus.cc:729] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d1f26e8554b868c2d771738935e07, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.714589 13549 consensus_queue.cc:260] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.714704 13551 tablet_bootstrap.cc:654] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.714795 13549 ts_tablet_manager.cc:1428] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.715823 13576 ts_tablet_manager.cc:1905] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:04.716243 12925 catalog_manager.cc:4841] TS 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829): tablet 0c302ba96e624bf091b1ad363e361c75 (table test-table [id=6ffd1a9daff04dddb3c15d60a76e6762]) successfully deleted
I20260810 15:21:04.716614 13551 tablet_bootstrap.cc:492] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: No bootstrap required, opened a new log
I20260810 15:21:04.716674 13551 ts_tablet_manager.cc:1397] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:21:04.716843 13551 raft_consensus.cc:348] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.716930 13551 raft_consensus.cc:374] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.716954 13551 raft_consensus.cc:729] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.717029 13551 consensus_queue.cc:260] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.717142 13551 ts_tablet_manager.cc:1428] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:21:04.717584 13578 tablet_replica.cc:330] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: stopping tablet replica
I20260810 15:21:04.717748 13578 raft_consensus.cc:2229] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:04.717880 13578 raft_consensus.cc:2258] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:04.718205 13576 ts_tablet_manager.cc:1918] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:04.718289 13576 log.cc:1196] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/0c302ba96e624bf091b1ad363e361c75
I20260810 15:21:04.718356 13578 ts_tablet_manager.cc:1905] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:04.718591 13576 ts_tablet_manager.cc:1939] T 0c302ba96e624bf091b1ad363e361c75 P d999b9cc80924aa5b59681d5573fe67f: Deleting consensus metadata
I20260810 15:21:04.718998 12925 catalog_manager.cc:4841] TS d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303): tablet 0c302ba96e624bf091b1ad363e361c75 (table test-table [id=6ffd1a9daff04dddb3c15d60a76e6762]) successfully deleted
I20260810 15:21:04.720501 13578 ts_tablet_manager.cc:1918] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:04.720564 13578 log.cc:1196] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/0c302ba96e624bf091b1ad363e361c75
I20260810 15:21:04.720829 13578 ts_tablet_manager.cc:1939] T 0c302ba96e624bf091b1ad363e361c75 P 052dbc5f756b477b84b75f9776655720: Deleting consensus metadata
I20260810 15:21:04.721225 12925 catalog_manager.cc:4841] TS 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509): tablet 0c302ba96e624bf091b1ad363e361c75 (table test-table [id=6ffd1a9daff04dddb3c15d60a76e6762]) successfully deleted
I20260810 15:21:04.764534 13559 raft_consensus.cc:482] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:04.764688 13559 raft_consensus.cc:504] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.765100 13559 leader_election.cc:288] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:04.769421 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.765744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:48075 (local address 127.4.64.67:37829)
0810 15:21:04.765795 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.765797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.765804 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:21:04.765829 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.765831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.765839 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.765911 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.766093 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.766695 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.768285 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.768914 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.768997 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.769011 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.769301 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.769344 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:21:04.769448 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.765571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37829 (local address 127.4.64.65:48075)
0810 15:21:04.765687 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.765696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.765739 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.765930 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.765931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.765933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.766016 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.766017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.766810 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.766811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.768187 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.768189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.769027 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.769028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.769105 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.769108 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.769288 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.769344 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":38,"thread_start_us":57,"threads_started":1}
I20260810 15:21:04.769711 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b237f0eb46d044b29d4f1a17465ef2d8" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07" is_pre_election: true
I20260810 15:21:04.769820 13465 raft_consensus.cc:2454] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 0.
I20260810 15:21:04.770004 13053 leader_election.cc:302] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, 5e9d1f26e8554b868c2d771738935e07; no voters:
I20260810 15:21:04.770072 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.765370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:47327 (local address 127.4.64.66:34303)
0810 15:21:04.765440 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.765443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.765545 (+ 102us) server_negotiation.cc:407] Connection header received
0810 15:21:04.765629 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.765630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.765640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.765717 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:04.765939 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.766770 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.769051 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.769657 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.769729 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.769740 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.769959 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.770001 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:21:04.770135 13559 raft_consensus.cc:2781] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:04.770174 13559 raft_consensus.cc:482] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:04.770202 13559 raft_consensus.cc:3037] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.770298 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.765352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:34303 (local address 127.4.64.65:47327)
0810 15:21:04.765471 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.765484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.765550 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.765777 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.765779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.765781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:04.765861 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.765863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.766864 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.766865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.768186 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.768189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.769795 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.769796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.769878 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.769882 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.770164 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.770217 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":71,"thread_start_us":73,"threads_started":1}
I20260810 15:21:04.771004 13559 raft_consensus.cc:504] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.771147 13559 leader_election.cc:288] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 election: Requested vote from peers d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:04.771414 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b237f0eb46d044b29d4f1a17465ef2d8" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07"
I20260810 15:21:04.771524 13465 raft_consensus.cc:3037] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.771543 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b237f0eb46d044b29d4f1a17465ef2d8" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d999b9cc80924aa5b59681d5573fe67f" is_pre_election: true
I20260810 15:21:04.771688 13292 raft_consensus.cc:2454] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 0.
I20260810 15:21:04.772289 13465 raft_consensus.cc:2454] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 1.
I20260810 15:21:04.772419 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b237f0eb46d044b29d4f1a17465ef2d8" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d999b9cc80924aa5b59681d5573fe67f"
I20260810 15:21:04.772480 13053 leader_election.cc:302] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, 5e9d1f26e8554b868c2d771738935e07; no voters:
I20260810 15:21:04.772511 13290 raft_consensus.cc:3037] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:04.772554 13559 raft_consensus.cc:2781] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:04.772677 13559 raft_consensus.cc:686] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 1 LEADER]: Becoming Leader. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Running, Role: LEADER
I20260810 15:21:04.772750 13559 consensus_queue.cc:237] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.773319 13290 raft_consensus.cc:2454] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 1.
I20260810 15:21:04.773595 12940 catalog_manager.cc:5495] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 052dbc5f756b477b84b75f9776655720 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } health_report { overall_health: UNKNOWN } } }
Aug 10 15:21:04 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375263, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.126@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:04.803098 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.794588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.4.64.126:34659)
0810 15:21:04.794644 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:04.794646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:04.794787 (+ 141us) server_negotiation.cc:407] Connection header received
0810 15:21:04.794855 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:04.794856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:04.794866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:04.794921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:04.795596 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.796118 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.797168 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.797370 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.797430 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.800110 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:04.800111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:04.800111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:04.800126 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:04.801741 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.802184 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.802185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.802186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.802248 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:04.802462 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:04.802463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:04.802463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:04.802590 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:04.802687 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:04.802856 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 15:21:04.802976 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:21:04.803210 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:04.794472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34659 (local address 127.0.0.1:37912)
0810 15:21:04.794681 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:04.794695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:04.794784 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:04.794954 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:04.794955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:04.795423 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:04.795491 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.795495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.796217 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.796218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.797067 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:04.797070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:04.797460 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:04.797462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:04.797526 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:04.798206 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:04.798213 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:04.799999 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:04.801852 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.801863 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.801865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.802081 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.802318 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:04.802320 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:04.802320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:04.802388 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:04.802680 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:04.802690 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:04.802740 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:21:04.802831 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:21:04.803039 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":90,"thread_start_us":108,"threads_started":1}
I20260810 15:21:04.805771 12940 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37912:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 15:21:04.806084 12940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:04.807837 13269 tablet_service.cc:1467] Processing CreateTablet for tablet 128b9379b3b343d6827776c7ba5343b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38f0dc42ebbe483097be7707df3307ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:21:04.807852 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 128b9379b3b343d6827776c7ba5343b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38f0dc42ebbe483097be7707df3307ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:21:04.807886 13437 tablet_service.cc:1467] Processing CreateTablet for tablet 128b9379b3b343d6827776c7ba5343b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38f0dc42ebbe483097be7707df3307ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:21:04.808020 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128b9379b3b343d6827776c7ba5343b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.808022 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128b9379b3b343d6827776c7ba5343b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.808054 13437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128b9379b3b343d6827776c7ba5343b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:04.809842 13551 tablet_bootstrap.cc:492] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: Bootstrap starting.
I20260810 15:21:04.810101 13550 tablet_bootstrap.cc:492] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: Bootstrap starting.
I20260810 15:21:04.810223 13551 tablet_bootstrap.cc:654] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.810614 13550 tablet_bootstrap.cc:654] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.811005 13551 tablet_bootstrap.cc:492] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: No bootstrap required, opened a new log
I20260810 15:21:04.811064 13551 ts_tablet_manager.cc:1397] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:04.811215 13551 raft_consensus.cc:348] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.811268 13549 tablet_bootstrap.cc:492] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: Bootstrap starting.
I20260810 15:21:04.811290 13551 raft_consensus.cc:374] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.811317 13551 raft_consensus.cc:729] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.811355 13551 consensus_queue.cc:260] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.811458 13551 ts_tablet_manager.cc:1428] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:21:04.811518 13584 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:04.811596 13584 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:21:04.811828 13549 tablet_bootstrap.cc:654] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:04.812561 13550 tablet_bootstrap.cc:492] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: No bootstrap required, opened a new log
I20260810 15:21:04.812621 13550 ts_tablet_manager.cc:1397] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 15:21:04.812769 13550 raft_consensus.cc:348] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.812855 13550 raft_consensus.cc:374] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.812876 13550 raft_consensus.cc:729] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.812923 13550 consensus_queue.cc:260] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.813015 13550 ts_tablet_manager.cc:1428] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 15:21:04.813125 13558 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:04.813197 13558 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:21:04.813594 13549 tablet_bootstrap.cc:492] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: No bootstrap required, opened a new log
I20260810 15:21:04.813692 13549 ts_tablet_manager.cc:1397] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:21:04.813838 13549 raft_consensus.cc:348] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.813922 13549 raft_consensus.cc:374] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:04.813953 13549 raft_consensus.cc:729] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d1f26e8554b868c2d771738935e07, State: Initialized, Role: FOLLOWER
I20260810 15:21:04.814009 13549 consensus_queue.cc:260] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:04.814113 13549 ts_tablet_manager.cc:1428] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:04.814203 13557 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:04.814314 13557 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 15:21:04.815129 13518 tablet.cc:2369] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:21:04.880151 13352 tablet.cc:2369] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:21:05.101132 13584 raft_consensus.cc:482] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:05.101255 13584 raft_consensus.cc:504] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.101555 13584 leader_election.cc:288] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:05.101856 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128b9379b3b343d6827776c7ba5343b0" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d999b9cc80924aa5b59681d5573fe67f" is_pre_election: true
I20260810 15:21:05.101892 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128b9379b3b343d6827776c7ba5343b0" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07" is_pre_election: true
I20260810 15:21:05.101960 13290 raft_consensus.cc:2454] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 0.
I20260810 15:21:05.101996 13465 raft_consensus.cc:2454] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 0.
I20260810 15:21:05.102154 13053 leader_election.cc:302] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:05.102319 13584 raft_consensus.cc:2781] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:05.102367 13584 raft_consensus.cc:482] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:05.102389 13584 raft_consensus.cc:3037] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.103163 13584 raft_consensus.cc:504] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.103314 13584 leader_election.cc:288] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 election: Requested vote from peers d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:05.103523 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128b9379b3b343d6827776c7ba5343b0" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d999b9cc80924aa5b59681d5573fe67f"
I20260810 15:21:05.103523 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128b9379b3b343d6827776c7ba5343b0" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07"
I20260810 15:21:05.103618 13290 raft_consensus.cc:3037] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.103616 13465 raft_consensus.cc:3037] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.104291 13465 raft_consensus.cc:2454] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 1.
I20260810 15:21:05.104310 13290 raft_consensus.cc:2454] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 1.
I20260810 15:21:05.104463 13053 leader_election.cc:302] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:05.104640 13584 raft_consensus.cc:2781] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:05.104771 13584 raft_consensus.cc:686] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [term 1 LEADER]: Becoming Leader. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Running, Role: LEADER
I20260810 15:21:05.104831 13584 consensus_queue.cc:237] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.105329 13559 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: TxnStatusTablet state changed. Reason: New leader 052dbc5f756b477b84b75f9776655720. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.105449 13559 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:21:05.105727 13598 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 15:21:05.105742 12940 catalog_manager.cc:5495] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 052dbc5f756b477b84b75f9776655720 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260810 15:21:05.105886 13598 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 15:21:05.170855 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.168160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.4.64.65:46509)
0810 15:21:05.168231 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.168234 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.168245 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:05.168338 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.168339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.168360 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.168425 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.168582 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.169089 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.169965 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.170180 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.170252 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.170476 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.170595 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.170676 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.170738 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.170770 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:21:05.170867 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.168090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46509 (local address 127.0.0.1:52354)
0810 15:21:05.168171 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:05.168185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:05.168241 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:05.168445 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:05.168446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:05.168447 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:21:05.168503 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.168506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.169178 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.169179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.169870 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.169872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.170287 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.170288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.170362 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.170379 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.170671 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.170683 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 15:21:05.170735 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:21:05.170794 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:21:05.172154 13465 raft_consensus.cc:1261] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Refusing update from remote peer 052dbc5f756b477b84b75f9776655720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:05.172250 13290 raft_consensus.cc:1261] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Refusing update from remote peer 052dbc5f756b477b84b75f9776655720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:05.172452 13584 consensus_queue.cc:1035] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.172578 13559 consensus_queue.cc:1035] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.173228 13557 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: TxnStatusTablet state changed. Reason: New leader 052dbc5f756b477b84b75f9776655720. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.173333 13557 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:21:05.173421 13558 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: TxnStatusTablet state changed. Reason: New leader 052dbc5f756b477b84b75f9776655720. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.173530 13558 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:21:05.174122 13559 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.174203 13584 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.174290 13584 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:21:05.174219 13559 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 052dbc5f756b477b84b75f9776655720: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:21:05.174654 13558 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.174741 13558 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P d999b9cc80924aa5b59681d5573fe67f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:21:05.174993 13557 tablet_replica.cc:439] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } } }
I20260810 15:21:05.175104 13557 tablet_replica.cc:442] T 128b9379b3b343d6827776c7ba5343b0 P 5e9d1f26e8554b868c2d771738935e07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:21:05.179343 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.176530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.4.64.65:46509)
0810 15:21:05.176596 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.176598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.176608 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:21:05.176631 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.176632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.176645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.176714 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.176852 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.177650 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.178497 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.178772 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.178851 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.179026 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.179064 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.179120 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.179239 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.179266 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:21:05.183821 13604 consensus_queue.cc:1035] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.185302 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.182159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:48273 (local address 127.4.64.65:46509)
0810 15:21:05.182228 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.182230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.182368 (+ 138us) server_negotiation.cc:407] Connection header received
0810 15:21:05.182429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.182429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.182440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.182494 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.182632 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.183161 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.184495 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.184746 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.184802 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.185027 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.185060 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.185089 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.185205 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.185228 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:21:05.185562 13559 consensus_queue.cc:1035] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.185694 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.182096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46509 (local address 127.4.64.65:48273)
0810 15:21:05.182296 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:05.182308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:05.182382 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:05.182497 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:05.182498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:05.182499 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:21:05.182555 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.182556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.183239 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.183241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.184248 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.184252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.184851 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.184852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.184922 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.184940 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.185143 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.185144 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:21:05.185430 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 15:21:05.185485 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
I20260810 15:21:05.199751 12939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:05.199913 12939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:05.200628 12939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Sending DeleteTablet for 3 replicas of tablet b237f0eb46d044b29d4f1a17465ef2d8
I20260810 15:21:05.200879 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet b237f0eb46d044b29d4f1a17465ef2d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:52340
I20260810 15:21:05.200930 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet b237f0eb46d044b29d4f1a17465ef2d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:50818
I20260810 15:21:05.200978 13578 tablet_replica.cc:330] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: stopping tablet replica
I20260810 15:21:05.201042 13578 raft_consensus.cc:2229] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:21:05.201049 13437 tablet_service.cc:1514] Processing DeleteTablet for tablet b237f0eb46d044b29d4f1a17465ef2d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:59598
I20260810 15:21:05.201154 13576 tablet_replica.cc:330] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: stopping tablet replica
I20260810 15:21:05.201153 13575 tablet_replica.cc:330] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: stopping tablet replica
I20260810 15:21:05.201215 13576 raft_consensus.cc:2229] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:05.201211 13575 raft_consensus.cc:2229] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:05.201227 13578 raft_consensus.cc:2258] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.201253 13576 raft_consensus.cc:2258] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.201260 13575 raft_consensus.cc:2258] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.201710 13575 ts_tablet_manager.cc:1905] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.201887 13576 ts_tablet_manager.cc:1905] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.204084 13578 ts_tablet_manager.cc:1905] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.205976 13578 ts_tablet_manager.cc:1918] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:21:05.206045 13578 log.cc:1196] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/b237f0eb46d044b29d4f1a17465ef2d8
I20260810 15:21:05.206284 13578 ts_tablet_manager.cc:1939] T b237f0eb46d044b29d4f1a17465ef2d8 P 052dbc5f756b477b84b75f9776655720: Deleting consensus metadata
I20260810 15:21:05.206516 13576 ts_tablet_manager.cc:1918] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:21:05.206585 13576 log.cc:1196] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/b237f0eb46d044b29d4f1a17465ef2d8
I20260810 15:21:05.206835 13576 ts_tablet_manager.cc:1939] T b237f0eb46d044b29d4f1a17465ef2d8 P d999b9cc80924aa5b59681d5573fe67f: Deleting consensus metadata
I20260810 15:21:05.207305 13575 ts_tablet_manager.cc:1918] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:21:05.207381 13575 log.cc:1196] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/b237f0eb46d044b29d4f1a17465ef2d8
I20260810 15:21:05.207592 13575 ts_tablet_manager.cc:1939] T b237f0eb46d044b29d4f1a17465ef2d8 P 5e9d1f26e8554b868c2d771738935e07: Deleting consensus metadata
I20260810 15:21:05.209029 12925 catalog_manager.cc:4841] TS 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509): tablet b237f0eb46d044b29d4f1a17465ef2d8 (table test-table [id=843a25cb84ba457893b42e1c5a2c79ad]) successfully deleted
I20260810 15:21:05.209143 12925 catalog_manager.cc:4841] TS 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829): tablet b237f0eb46d044b29d4f1a17465ef2d8 (table test-table [id=843a25cb84ba457893b42e1c5a2c79ad]) successfully deleted
I20260810 15:21:05.209192 12925 catalog_manager.cc:4841] TS d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303): tablet b237f0eb46d044b29d4f1a17465ef2d8 (table test-table [id=843a25cb84ba457893b42e1c5a2c79ad]) successfully deleted
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[12887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:21:05.230080 13000 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:05.223844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.4.64.126:34659)
0810 15:21:05.223913 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.223915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.224023 (+ 108us) server_negotiation.cc:407] Connection header received
0810 15:21:05.224100 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.224101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.224114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.224204 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:05.224822 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.225694 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.226537 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.226866 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.226948 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.230012 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37928: BlockingRecv error: recv got EOF from 127.0.0.1:37928 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[12887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM
I20260810 15:21:05.238827 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.230980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.4.64.126:34659)
0810 15:21:05.231020 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.231022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.231114 (+ 92us) server_negotiation.cc:407] Connection header received
0810 15:21:05.231144 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.231145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.231155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.231228 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:05.231659 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.232405 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.233089 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.233364 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.233443 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.235720 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:05.235721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:05.235722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:05.235730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:05.237396 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:05.237836 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:05.237837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:05.237838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:05.237913 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:05.238169 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:05.238170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:05.238170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:05.238356 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:05.238472 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.238624 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.238723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:21:05.240749 12939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:21:05.240947 12939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:05.242548 13437 tablet_service.cc:1467] Processing CreateTablet for tablet e31ca018eb9a42878eaca3760847e1d6 (DEFAULT_TABLE table=test-table [id=85288c3b3eab43878e637924018acabd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:05.242548 13103 tablet_service.cc:1467] Processing CreateTablet for tablet e31ca018eb9a42878eaca3760847e1d6 (DEFAULT_TABLE table=test-table [id=85288c3b3eab43878e637924018acabd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:05.242551 13269 tablet_service.cc:1467] Processing CreateTablet for tablet e31ca018eb9a42878eaca3760847e1d6 (DEFAULT_TABLE table=test-table [id=85288c3b3eab43878e637924018acabd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:05.242705 13437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e31ca018eb9a42878eaca3760847e1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:05.242705 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e31ca018eb9a42878eaca3760847e1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:05.242712 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e31ca018eb9a42878eaca3760847e1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:05.244086 13549 tablet_bootstrap.cc:492] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Bootstrap starting.
I20260810 15:21:05.244114 13551 tablet_bootstrap.cc:492] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Bootstrap starting.
I20260810 15:21:05.244215 13550 tablet_bootstrap.cc:492] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Bootstrap starting.
I20260810 15:21:05.244546 13549 tablet_bootstrap.cc:654] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:05.244546 13551 tablet_bootstrap.cc:654] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:05.244601 13550 tablet_bootstrap.cc:654] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:05.245291 13549 tablet_bootstrap.cc:492] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: No bootstrap required, opened a new log
I20260810 15:21:05.245371 13549 ts_tablet_manager.cc:1397] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:05.245397 13550 tablet_bootstrap.cc:492] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: No bootstrap required, opened a new log
I20260810 15:21:05.245458 13550 ts_tablet_manager.cc:1397] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:05.245527 13549 raft_consensus.cc:348] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.245651 13549 raft_consensus.cc:374] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:05.245573 13550 raft_consensus.cc:348] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.245683 13550 raft_consensus.cc:374] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:05.245685 13549 raft_consensus.cc:729] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d1f26e8554b868c2d771738935e07, State: Initialized, Role: FOLLOWER
I20260810 15:21:05.245713 13550 raft_consensus.cc:729] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Initialized, Role: FOLLOWER
I20260810 15:21:05.245744 13549 consensus_queue.cc:260] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.245766 13550 consensus_queue.cc:260] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.245842 13549 ts_tablet_manager.cc:1428] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:05.245850 13550 ts_tablet_manager.cc:1428] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:21:05.246351 13551 tablet_bootstrap.cc:492] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: No bootstrap required, opened a new log
I20260810 15:21:05.246407 13551 ts_tablet_manager.cc:1397] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:21:05.246528 13551 raft_consensus.cc:348] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.246608 13551 raft_consensus.cc:374] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:05.246630 13551 raft_consensus.cc:729] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Initialized, Role: FOLLOWER
I20260810 15:21:05.246675 13551 consensus_queue.cc:260] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.246785 13551 ts_tablet_manager.cc:1428] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:05.585850 13584 raft_consensus.cc:482] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:05.585974 13584 raft_consensus.cc:504] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.586110 13584 leader_election.cc:288] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:05.586401 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31ca018eb9a42878eaca3760847e1d6" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d999b9cc80924aa5b59681d5573fe67f" is_pre_election: true
I20260810 15:21:05.586517 13290 raft_consensus.cc:2454] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 0.
I20260810 15:21:05.586460 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31ca018eb9a42878eaca3760847e1d6" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07" is_pre_election: true
I20260810 15:21:05.586602 13465 raft_consensus.cc:2454] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 0.
I20260810 15:21:05.586719 13053 leader_election.cc:302] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:05.586833 13584 raft_consensus.cc:2781] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:05.586895 13584 raft_consensus.cc:482] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:05.586913 13584 raft_consensus.cc:3037] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.587785 13584 raft_consensus.cc:504] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.587949 13584 leader_election.cc:288] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 election: Requested vote from peers d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:05.588148 13290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31ca018eb9a42878eaca3760847e1d6" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d999b9cc80924aa5b59681d5573fe67f"
I20260810 15:21:05.588186 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e31ca018eb9a42878eaca3760847e1d6" candidate_uuid: "052dbc5f756b477b84b75f9776655720" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07"
I20260810 15:21:05.588253 13290 raft_consensus.cc:3037] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.588310 13465 raft_consensus.cc:3037] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.589178 13465 raft_consensus.cc:2454] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 1.
I20260810 15:21:05.589178 13290 raft_consensus.cc:2454] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 052dbc5f756b477b84b75f9776655720 in term 1.
I20260810 15:21:05.589419 13053 leader_election.cc:302] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, 5e9d1f26e8554b868c2d771738935e07; no voters:
I20260810 15:21:05.589531 13584 raft_consensus.cc:2781] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:05.589649 13584 raft_consensus.cc:686] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 1 LEADER]: Becoming Leader. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Running, Role: LEADER
I20260810 15:21:05.589730 13584 consensus_queue.cc:237] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.590410 12935 catalog_manager.cc:5495] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 reported cstate change: term changed from 0 to 1, leader changed from <none> to 052dbc5f756b477b84b75f9776655720 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "052dbc5f756b477b84b75f9776655720" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260810 15:21:05.604542 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.601972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.4.64.65:46509)
0810 15:21:05.602035 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.602036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.602044 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:05.602067 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.602068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.602077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.602119 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.602366 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.602862 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.603743 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.603949 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.604006 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.604277 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.604309 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.604348 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.604419 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.604437 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:21:05.605563 13465 raft_consensus.cc:1261] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Refusing update from remote peer 052dbc5f756b477b84b75f9776655720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:05.605587 13290 raft_consensus.cc:1261] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Refusing update from remote peer 052dbc5f756b477b84b75f9776655720: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:05.605875 13559 consensus_queue.cc:1035] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.605984 13584 consensus_queue.cc:1035] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.608675 12939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:05.608790 12939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:05.609544 12939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Sending DeleteTablet for 3 replicas of tablet e31ca018eb9a42878eaca3760847e1d6
I20260810 15:21:05.609932 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet e31ca018eb9a42878eaca3760847e1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:50818
I20260810 15:21:05.609941 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet e31ca018eb9a42878eaca3760847e1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:52340
I20260810 15:21:05.610075 13578 tablet_replica.cc:330] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: stopping tablet replica
I20260810 15:21:05.610081 13437 tablet_service.cc:1514] Processing DeleteTablet for tablet e31ca018eb9a42878eaca3760847e1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:59598
I20260810 15:21:05.610148 13578 raft_consensus.cc:2229] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:21:05.610176 13575 tablet_replica.cc:330] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: stopping tablet replica
I20260810 15:21:05.610183 13576 tablet_replica.cc:330] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: stopping tablet replica
I20260810 15:21:05.610210 13578 raft_consensus.cc:2258] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.610229 13575 raft_consensus.cc:2229] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:05.610240 13576 raft_consensus.cc:2229] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:05.610277 13575 raft_consensus.cc:2258] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.610287 13576 raft_consensus.cc:2258] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.610316 12939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:21:05.610484 12939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:05.610589 13578 ts_tablet_manager.cc:1905] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.610809 13576 ts_tablet_manager.cc:1905] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.611738 13103 tablet_service.cc:1467] Processing CreateTablet for tablet 43f94033e0914ed78c6188233063dd3f (DEFAULT_TABLE table=test-table [id=adf3bfc861434d5689887aec96d99c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:05.611775 13437 tablet_service.cc:1467] Processing CreateTablet for tablet 43f94033e0914ed78c6188233063dd3f (DEFAULT_TABLE table=test-table [id=adf3bfc861434d5689887aec96d99c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:05.611868 13103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f94033e0914ed78c6188233063dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:05.611876 13437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f94033e0914ed78c6188233063dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:05.612407 13575 ts_tablet_manager.cc:1905] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.612574 13578 ts_tablet_manager.cc:1918] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:05.612632 13578 log.cc:1196] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/e31ca018eb9a42878eaca3760847e1d6
I20260810 15:21:05.612601 13576 ts_tablet_manager.cc:1918] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:05.612674 13576 log.cc:1196] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/e31ca018eb9a42878eaca3760847e1d6
I20260810 15:21:05.612869 13578 ts_tablet_manager.cc:1939] T e31ca018eb9a42878eaca3760847e1d6 P 052dbc5f756b477b84b75f9776655720: Deleting consensus metadata
I20260810 15:21:05.612936 13576 ts_tablet_manager.cc:1939] T e31ca018eb9a42878eaca3760847e1d6 P d999b9cc80924aa5b59681d5573fe67f: Deleting consensus metadata
I20260810 15:21:05.613466 13549 tablet_bootstrap.cc:492] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Bootstrap starting.
I20260810 15:21:05.613927 12925 catalog_manager.cc:4841] TS d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303): tablet e31ca018eb9a42878eaca3760847e1d6 (table test-table [id=85288c3b3eab43878e637924018acabd]) successfully deleted
I20260810 15:21:05.613996 12925 catalog_manager.cc:4841] TS 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509): tablet e31ca018eb9a42878eaca3760847e1d6 (table test-table [id=85288c3b3eab43878e637924018acabd]) successfully deleted
I20260810 15:21:05.613992 13549 tablet_bootstrap.cc:654] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:05.614070 13575 ts_tablet_manager.cc:1918] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:05.614137 13575 log.cc:1196] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/e31ca018eb9a42878eaca3760847e1d6
I20260810 15:21:05.614269 13551 tablet_bootstrap.cc:492] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720: Bootstrap starting.
I20260810 15:21:05.614382 13575 ts_tablet_manager.cc:1939] T e31ca018eb9a42878eaca3760847e1d6 P 5e9d1f26e8554b868c2d771738935e07: Deleting consensus metadata
I20260810 15:21:05.614562 13269 tablet_service.cc:1467] Processing CreateTablet for tablet 43f94033e0914ed78c6188233063dd3f (DEFAULT_TABLE table=test-table [id=adf3bfc861434d5689887aec96d99c6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:05.614606 13551 tablet_bootstrap.cc:654] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:05.614671 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f94033e0914ed78c6188233063dd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:05.614945 13549 tablet_bootstrap.cc:492] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: No bootstrap required, opened a new log
I20260810 15:21:05.614998 13549 ts_tablet_manager.cc:1397] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:21:05.615144 13551 tablet_bootstrap.cc:492] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720: No bootstrap required, opened a new log
I20260810 15:21:05.615128 13549 raft_consensus.cc:348] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.615178 13551 ts_tablet_manager.cc:1397] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:05.615195 13549 raft_consensus.cc:374] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:05.615216 13549 raft_consensus.cc:729] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9d1f26e8554b868c2d771738935e07, State: Initialized, Role: FOLLOWER
I20260810 15:21:05.615267 13549 consensus_queue.cc:260] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.615283 13551 raft_consensus.cc:348] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.615358 13551 raft_consensus.cc:374] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:05.615362 13549 ts_tablet_manager.cc:1428] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:05.615381 13551 raft_consensus.cc:729] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 052dbc5f756b477b84b75f9776655720, State: Initialized, Role: FOLLOWER
I20260810 15:21:05.615420 13551 consensus_queue.cc:260] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.615535 13551 ts_tablet_manager.cc:1428] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:21:05.616024 13550 tablet_bootstrap.cc:492] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Bootstrap starting.
I20260810 15:21:05.616223 12925 catalog_manager.cc:4841] TS 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829): tablet e31ca018eb9a42878eaca3760847e1d6 (table test-table [id=85288c3b3eab43878e637924018acabd]) successfully deleted
I20260810 15:21:05.616385 13550 tablet_bootstrap.cc:654] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:05.617018 13550 tablet_bootstrap.cc:492] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: No bootstrap required, opened a new log
I20260810 15:21:05.617065 13550 ts_tablet_manager.cc:1397] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:05.617228 13550 raft_consensus.cc:348] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.617285 13550 raft_consensus.cc:374] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:05.617326 13550 raft_consensus.cc:729] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Initialized, Role: FOLLOWER
I20260810 15:21:05.617376 13550 consensus_queue.cc:260] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.617466 13550 ts_tablet_manager.cc:1428] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:05.709571 13561 raft_consensus.cc:482] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:05.709735 13561 raft_consensus.cc:504] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.709925 13561 leader_election.cc:288] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:05.710173 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f94033e0914ed78c6188233063dd3f" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052dbc5f756b477b84b75f9776655720" is_pre_election: true
I20260810 15:21:05.710281 13130 raft_consensus.cc:2454] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 0.
I20260810 15:21:05.710233 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f94033e0914ed78c6188233063dd3f" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07" is_pre_election: true
I20260810 15:21:05.710412 13465 raft_consensus.cc:2454] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 0.
I20260810 15:21:05.710513 13216 leader_election.cc:302] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:05.710644 13561 raft_consensus.cc:2781] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:05.710696 13561 raft_consensus.cc:482] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:05.710726 13561 raft_consensus.cc:3037] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.711524 13561 raft_consensus.cc:504] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.711669 13561 leader_election.cc:288] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 election: Requested vote from peers 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509), 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829)
I20260810 15:21:05.711844 13130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f94033e0914ed78c6188233063dd3f" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "052dbc5f756b477b84b75f9776655720"
I20260810 15:21:05.711921 13130 raft_consensus.cc:3037] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.711897 13465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f94033e0914ed78c6188233063dd3f" candidate_uuid: "d999b9cc80924aa5b59681d5573fe67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9d1f26e8554b868c2d771738935e07"
I20260810 15:21:05.711993 13465 raft_consensus.cc:3037] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:05.712584 13130 raft_consensus.cc:2454] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 1.
I20260810 15:21:05.712668 13465 raft_consensus.cc:2454] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d999b9cc80924aa5b59681d5573fe67f in term 1.
I20260810 15:21:05.712814 13216 leader_election.cc:302] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 052dbc5f756b477b84b75f9776655720, d999b9cc80924aa5b59681d5573fe67f; no voters:
I20260810 15:21:05.712936 13561 raft_consensus.cc:2781] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:05.713065 13561 raft_consensus.cc:686] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 1 LEADER]: Becoming Leader. State: Replica: d999b9cc80924aa5b59681d5573fe67f, State: Running, Role: LEADER
I20260810 15:21:05.713124 13561 consensus_queue.cc:237] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } }
I20260810 15:21:05.713855 12939 catalog_manager.cc:5495] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f reported cstate change: term changed from 0 to 1, leader changed from <none> to d999b9cc80924aa5b59681d5573fe67f (127.4.64.66). New cstate: current_term: 1 leader_uuid: "d999b9cc80924aa5b59681d5573fe67f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d999b9cc80924aa5b59681d5573fe67f" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 } health_report { overall_health: UNKNOWN } } }
I20260810 15:21:05.738397 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.735581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.4.64.66:34303)
0810 15:21:05.735727 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.735729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.735736 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:21:05.735764 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.735765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.735776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.735853 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.735989 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.736590 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.737412 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.737703 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.737793 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.738092 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.738124 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.738193 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.738263 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.738291 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260810 15:21:05.743902 13547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.741051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:54529 (local address 127.4.64.65:46509)
0810 15:21:05.741130 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.741132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.741275 (+ 143us) server_negotiation.cc:407] Connection header received
0810 15:21:05.741350 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.741351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.741364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.741420 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.741527 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.742132 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.743014 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.743249 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.743331 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.743542 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.743582 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.743666 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.743732 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.743793 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:21:05.743906 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.740961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46509 (local address 127.4.64.66:54529)
0810 15:21:05.741186 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:05.741199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:05.741292 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:05.741421 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:05.741422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:05.741423 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:21:05.741470 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.741472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.742284 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.742286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.742914 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.742916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.743350 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.743352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.743414 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.743437 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.743669 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.743672 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:05.743725 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:21:05.743805 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260810 15:21:05.746105 13465 raft_consensus.cc:1261] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Refusing update from remote peer d999b9cc80924aa5b59681d5573fe67f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:05.746142 13130 raft_consensus.cc:1261] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Refusing update from remote peer d999b9cc80924aa5b59681d5573fe67f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:05.746374 13561 consensus_queue.cc:1035] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9d1f26e8554b868c2d771738935e07" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.746496 13638 consensus_queue.cc:1035] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [LEADER]: Connected to new peer: Peer: permanent_uuid: "052dbc5f756b477b84b75f9776655720" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:05.754048 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.751407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:45183 (local address 127.4.64.66:34303)
0810 15:21:05.751446 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:05.751448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:05.751455 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:21:05.751478 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:05.751479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:05.751488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:05.751540 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:05.751677 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.752247 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.753116 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.753401 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.753516 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.753709 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.753760 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.753831 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:05.753936 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 15:21:05.753961 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:21:05.754074 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.751210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:34303 (local address 127.4.64.65:45183)
0810 15:21:05.751339 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:05.751352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:05.751404 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:05.751548 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:05.751549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:05.751552 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:21:05.751612 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.751613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.752344 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.752346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.753008 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:05.753010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:05.753518 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:05.753519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:05.753592 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:05.753598 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:21:05.753867 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:21:05.753869 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:21:05.753930 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:21:05.753989 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260810 15:21:05.763500 12939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:05.763638 12939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:05.764390 12939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 529105e32f98436390a1c239efd92f6e: Sending DeleteTablet for 3 replicas of tablet 43f94033e0914ed78c6188233063dd3f
I20260810 15:21:05.764778 13103 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f94033e0914ed78c6188233063dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:52340
I20260810 15:21:05.764863 13437 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f94033e0914ed78c6188233063dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:59598
I20260810 15:21:05.764859 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f94033e0914ed78c6188233063dd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:05 UTC) from {username='oryx', principal='oryx/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:50818
I20260810 15:21:05.764959 13575 tablet_replica.cc:330] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: stopping tablet replica
I20260810 15:21:05.764984 13578 tablet_replica.cc:330] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720: stopping tablet replica
I20260810 15:21:05.765008 13575 raft_consensus.cc:2229] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:05.765043 13575 raft_consensus.cc:2258] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.765036 13578 raft_consensus.cc:2229] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:05.765076 13578 raft_consensus.cc:2258] T 43f94033e0914ed78c6188233063dd3f P 052dbc5f756b477b84b75f9776655720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.765162 13576 tablet_replica.cc:330] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: stopping tablet replica
I20260810 15:21:05.765147 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 13027
I20260810 15:21:05.765226 13576 raft_consensus.cc:2229] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:21:05.765297 13576 raft_consensus.cc:2258] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:05.765451 13575 ts_tablet_manager.cc:1905] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.767657 13575 ts_tablet_manager.cc:1918] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:21:05.767745 13575 log.cc:1196] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/43f94033e0914ed78c6188233063dd3f
I20260810 15:21:05.767920 13576 ts_tablet_manager.cc:1905] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:05.768004 13575 ts_tablet_manager.cc:1939] T 43f94033e0914ed78c6188233063dd3f P 5e9d1f26e8554b868c2d771738935e07: Deleting consensus metadata
I20260810 15:21:05.769640 13576 ts_tablet_manager.cc:1918] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:21:05.769699 13576 log.cc:1196] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/43f94033e0914ed78c6188233063dd3f
I20260810 15:21:05.769811 12925 catalog_manager.cc:4841] TS 5e9d1f26e8554b868c2d771738935e07 (127.4.64.67:37829): tablet 43f94033e0914ed78c6188233063dd3f (table test-table [id=adf3bfc861434d5689887aec96d99c6b]) successfully deleted
I20260810 15:21:05.769891 13576 ts_tablet_manager.cc:1939] T 43f94033e0914ed78c6188233063dd3f P d999b9cc80924aa5b59681d5573fe67f: Deleting consensus metadata
I20260810 15:21:05.770463 12925 catalog_manager.cc:4841] TS d999b9cc80924aa5b59681d5573fe67f (127.4.64.66:34303): tablet 43f94033e0914ed78c6188233063dd3f (table test-table [id=adf3bfc861434d5689887aec96d99c6b]) successfully deleted
W20260810 15:21:05.777163 12925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:46509 (error 108)
I20260810 15:21:05.777561 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 13195
I20260810 15:21:05.777844 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:05.777574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:46509 (local address 127.0.0.1:52386)
0810 15:21:05.777726 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:05.777783 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:46509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
W20260810 15:21:05.779803 12925 catalog_manager.cc:4584] TS 052dbc5f756b477b84b75f9776655720 (127.4.64.65:46509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43f94033e0914ed78c6188233063dd3f: Network error: Client connection negotiation failed: client connection to 127.4.64.65:46509: connect: Connection refused (error 111)
I20260810 15:21:05.791541 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 13360
I20260810 15:21:05.802160 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 12903
2026-08-10T15:21:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3319 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[13659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[13659](info): set up 2 sockets
Aug 10 15:21:05 dist-test-slave-9zdj krb5kdc[13659](info): commencing operation
krb5kdc: starting...
W20260810 15:21:06.921584 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:21:07 dist-test-slave-9zdj krb5kdc[13659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:21:07.057030 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35947
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:35947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:07.142009 13674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:07.142283 13674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:07.142326 13674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:07.143534 13674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:07.143617 13674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:07.143641 13674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:07.143663 13674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:07.143693 13674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:07.145326 13674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:35947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:07.145659 13674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:07.145907 13674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:07.151225 13682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:07.151239 13684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:07.151239 13680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:07.151376 13674 server_base.cc:1034] running on GCE node
I20260810 15:21:07.151906 13674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:07.152230 13674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:07.153401 13674 hybrid_clock.cc:648] HybridClock initialized: now 1786375267153394 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T15:21:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1359 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:21:07 dist-test-slave-9zdj krb5kdc[13691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:21:07 dist-test-slave-9zdj krb5kdc[13691](info): set up 2 sockets
Aug 10 15:21:07 dist-test-slave-9zdj krb5kdc[13691](info): commencing operation
krb5kdc: starting...
W20260810 15:21:08.305251 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:21:08.450390 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34549
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:08.536712 13707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:08.536950 13707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:08.536975 13707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:08.538213 13707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:08.538262 13707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:08.538277 13707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:08.538290 13707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:08.538354 13707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:08.539894 13707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:08.540184 13707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:08.540436 13707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:08.545831 13713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:08.545915 13712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:08.545957 13707 server_base.cc:1034] running on GCE node
W20260810 15:21:08.545928 13715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:08.546336 13707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:08.546656 13707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:08.547881 13707 hybrid_clock.cc:648] HybridClock initialized: now 1786375268547855 us; error 59 us; skew 500 ppm
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:08.550613 13707 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:21:08.551033 13707 webserver.cc:460] Webserver started at http://127.4.64.126:46705/ using document root <none> and password file <none>
I20260810 15:21:08.551234 13707 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:08.551275 13707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:08.551370 13707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:08.552424 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "68573456fa6a4814b633a31019280214"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "2180c037c19584a5061af5e7771e29d9"
server_key_iv: "2f5246947a232321da16a6a4819d29e9"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.552744 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "68573456fa6a4814b633a31019280214"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "2180c037c19584a5061af5e7771e29d9"
server_key_iv: "2f5246947a232321da16a6a4819d29e9"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.554165 13707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:08.554963 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.555184 13707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:08.555275 13707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "68573456fa6a4814b633a31019280214"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "2180c037c19584a5061af5e7771e29d9"
server_key_iv: "2f5246947a232321da16a6a4819d29e9"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.555333 13707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:08.562752 13707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:08.564962 13707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:08.565091 13707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:08.569574 13707 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34549
I20260810 15:21:08.569671 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34549 every 8 connection(s)
I20260810 15:21:08.570086 13707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:21:08.571079 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:08.573493 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Bootstrap starting.
I20260810 15:21:08.574134 13789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:08.574440 13789 log.cc:824] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:08.575161 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: No bootstrap required, opened a new log
I20260810 15:21:08.576556 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 13707
I20260810 15:21:08.576689 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:21:08.577000 4353 external_mini_cluster.cc:1428] Setting key 0baaea1debbfae8f2c30dfcd5d3403f3
I20260810 15:21:08.578691 13789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:08.578841 13789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:08.578866 13789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68573456fa6a4814b633a31019280214, State: Initialized, Role: FOLLOWER
I20260810 15:21:08.579001 13789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:08.579075 13789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:21:08.579119 13789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:21:08.579209 13789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:08.580026 13789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:08.580121 13789 leader_election.cc:302] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68573456fa6a4814b633a31019280214; no voters:
I20260810 15:21:08.580384 13789 leader_election.cc:288] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:21:08.580410 13794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:08.580528 13794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 LEADER]: Becoming Leader. State: Replica: 68573456fa6a4814b633a31019280214, State: Running, Role: LEADER
I20260810 15:21:08.580613 13794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:08.580672 13789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:21:08.580964 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68573456fa6a4814b633a31019280214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } } }
I20260810 15:21:08.581038 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:08.581200 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68573456fa6a4814b633a31019280214. Latest consensus state: current_term: 1 leader_uuid: "68573456fa6a4814b633a31019280214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } } }
I20260810 15:21:08.581360 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:08.581563 13798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:08.582245 13798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:08.584179 13798 catalog_manager.cc:1347] Generated new cluster ID: ceaf011109f84d9d811ae62e72421bbb
I20260810 15:21:08.584244 13798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:08.590621 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:08.577753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.4.64.126:34549)
0810 15:21:08.577984 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:08.577996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:08.578028 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:21:08.578881 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:08.578891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:08.578962 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:08.579158 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:08.580005 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.581445 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.582710 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.583349 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.583467 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:08.586475 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:08.586480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:08.586480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:08.586493 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:08.588556 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:08.588994 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:08.588998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:08.588998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:08.589136 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:08.589372 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:08.589374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:08.589374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:08.589791 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:08.590022 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:08.590123 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 15:21:08.590388 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":122,"threads_started":1}
I20260810 15:21:08.593545 13798 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:21:08.594097 13798 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:08.598063 13798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Generated new TSK 0
I20260810 15:21:08.598313 13798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:21:08.637776 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34549
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:08.731135 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:08.731364 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:08.731388 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:08.732640 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:08.732682 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:08.732709 13820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:21:08.734184 13820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:08.734409 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:08.734602 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:08.739854 13826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:08.739876 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:08.739861 13827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:08.740660 13820 server_base.cc:1034] running on GCE node
I20260810 15:21:08.740856 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:08.741061 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:08.742206 13820 hybrid_clock.cc:648] HybridClock initialized: now 1786375268742178 us; error 49 us; skew 500 ppm
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:08.744823 13820 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:21:08.745313 13820 webserver.cc:460] Webserver started at http://127.4.64.65:33311/ using document root <none> and password file <none>
I20260810 15:21:08.745517 13820 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:08.745560 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:08.745719 13820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:08.746764 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "430e4ffad5f54a4b8c8d4b3b4875ac78"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "6d2a39720574249e2bd09d3df90e6624"
server_key_iv: "67eb39d716900893e61abdd5c8c86973"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.747080 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "430e4ffad5f54a4b8c8d4b3b4875ac78"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "6d2a39720574249e2bd09d3df90e6624"
server_key_iv: "67eb39d716900893e61abdd5c8c86973"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.748436 13820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:08.749161 13837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.749364 13820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:08.749475 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "430e4ffad5f54a4b8c8d4b3b4875ac78"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "6d2a39720574249e2bd09d3df90e6624"
server_key_iv: "67eb39d716900893e61abdd5c8c86973"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.749553 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:08.764366 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:08.767014 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:08.767204 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:08.767472 13820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:08.767841 13820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:08.767899 13820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.767941 13820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:08.767985 13820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.774374 13820 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:42937
I20260810 15:21:08.774438 13966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:42937 every 8 connection(s)
I20260810 15:21:08.774812 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:08.785166 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 13820
I20260810 15:21:08.785390 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:21:08.785683 4353 external_mini_cluster.cc:1428] Setting key 470013582f5e0eb401fab717d3244c0e
I20260810 15:21:08.785877 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:08.775715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:33773 (local address 127.4.64.126:34549)
0810 15:21:08.775801 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:08.775803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:08.775873 (+ 70us) server_negotiation.cc:407] Connection header received
0810 15:21:08.776397 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:08.776398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:08.776409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:08.776482 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:08.777540 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.778118 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.779534 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.779793 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.779861 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:08.782750 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:08.782752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:08.782752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:08.782759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:08.784136 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:08.784620 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:08.784621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:08.784622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:08.784679 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:08.784956 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:08.784958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:08.784958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:08.785137 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:08.785216 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:08.785592 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 15:21:08.785777 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 15:21:08.786063 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:08.775570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:33773)
0810 15:21:08.775745 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:08.775791 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:08.776284 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:08.776536 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:08.776540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:08.777131 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:08.777391 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:08.777396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.778443 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.778445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:08.779393 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:08.779396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.779889 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.779890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:08.780005 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:08.780787 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:08.780799 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:08.782627 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:08.784250 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:08.784257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:08.784259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:08.784516 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:08.784768 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:08.784768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:08.784768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:08.784882 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:08.785213 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:08.785219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:08.785418 (+ 199us) client_negotiation.cc:769] Sending connection context
0810 15:21:08.785539 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 15:21:08.785783 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260810 15:21:08.786624 13967 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34549
I20260810 15:21:08.786726 13967 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:08.786949 13967 heartbeater.cc:511] Master 127.4.64.126:34549 requested a full tablet report, sending...
I20260810 15:21:08.787591 13745 ts_manager.cc:194] Registered new tserver with Master: 430e4ffad5f54a4b8c8d4b3b4875ac78 (127.4.64.65:42937)
I20260810 15:21:08.788537 13745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:33773
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:21:08.817761 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34549
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:08.906508 13972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:08.906782 13972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:08.906822 13972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:08.908069 13972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:08.908149 13972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:08.908190 13972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:21:08.909745 13972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:08.910038 13972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:08.910305 13972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:08.915901 13977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:08.915913 13978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:08.915913 13980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:08.916167 13972 server_base.cc:1034] running on GCE node
I20260810 15:21:08.916417 13972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:08.916702 13972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:08.917894 13972 hybrid_clock.cc:648] HybridClock initialized: now 1786375268917876 us; error 50 us; skew 500 ppm
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:08.920599 13972 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:21:08.921152 13972 webserver.cc:460] Webserver started at http://127.4.64.66:33389/ using document root <none> and password file <none>
I20260810 15:21:08.921427 13972 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:08.921499 13972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:08.921701 13972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:08.922880 13972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "de914d9907c04cefac02fd6d013a3f95"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "279619a0f4a0e2bc7ec074de03759120"
server_key_iv: "18aa1b19c308e1b5f893877fae47bbba"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.923281 13972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "de914d9907c04cefac02fd6d013a3f95"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "279619a0f4a0e2bc7ec074de03759120"
server_key_iv: "18aa1b19c308e1b5f893877fae47bbba"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.924777 13972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:21:08.925856 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.926034 13972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:08.926118 13972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "de914d9907c04cefac02fd6d013a3f95"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "279619a0f4a0e2bc7ec074de03759120"
server_key_iv: "18aa1b19c308e1b5f893877fae47bbba"
server_key_version: "encryptionkey@0"
I20260810 15:21:08.926177 13972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:08.950728 13972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:08.953115 13972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:08.953258 13972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:08.953521 13972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:08.953954 13972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:08.953998 13972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.954023 13972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:08.954036 13972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:08.960633 13972 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:34233
I20260810 15:21:08.960680 14128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:34233 every 8 connection(s)
I20260810 15:21:08.961059 13972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:21:08.964474 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 13972
I20260810 15:21:08.964624 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:21:08.964839 4353 external_mini_cluster.cc:1428] Setting key 0dbc338ade8ac89654ea5ef4295fbb0a
Aug 10 15:21:08 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375268, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
I20260810 15:21:08.977638 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:08.961889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:58523 (local address 127.4.64.126:34549)
0810 15:21:08.961970 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:08.961973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:08.962252 (+ 279us) server_negotiation.cc:407] Connection header received
0810 15:21:08.962829 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:08.962830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:08.962842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:08.962908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:08.963978 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.964494 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.965922 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.966113 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.966192 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:08.974420 (+ 8228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:08.974423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:08.974423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:08.974433 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:08.975898 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:08.976405 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:08.976407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:08.976408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:08.976474 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:08.976782 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:08.976783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:08.976784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:08.976967 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:08.977038 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:08.977340 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 15:21:08.977504 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
I20260810 15:21:08.977825 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:08.961907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:58523)
0810 15:21:08.962085 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:08.962144 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:08.962709 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:08.962995 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:08.962999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:08.963638 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:08.963861 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:08.963867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.964609 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.964611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:08.965801 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:08.965804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:08.966204 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:08.966205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:08.966353 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:08.972389 (+ 6036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:08.972402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:08.974256 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:08.976011 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:08.976023 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:08.976026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:08.976297 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:08.976548 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:08.976549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:08.976550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:08.976683 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:08.977044 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:08.977051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:08.977200 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 15:21:08.977296 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 15:21:08.977469 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260810 15:21:08.978354 14129 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34549
I20260810 15:21:08.978464 14129 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:08.978677 14129 heartbeater.cc:511] Master 127.4.64.126:34549 requested a full tablet report, sending...
I20260810 15:21:08.979141 13745 ts_manager.cc:194] Registered new tserver with Master: de914d9907c04cefac02fd6d013a3f95 (127.4.64.66:34233)
I20260810 15:21:08.979555 13745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:58523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:21:08.997848 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34549
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:09.089574 14137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:09.089963 14137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:09.090008 14137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:09.091296 14137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:09.091419 14137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:09.091468 14137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:21:09.093089 14137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:09.093398 14137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:09.093704 14137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:09.099121 14147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.099277 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.099277 14145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.099829 14137 server_base.cc:1034] running on GCE node
I20260810 15:21:09.100024 14137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:09.100322 14137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:09.101519 14137 hybrid_clock.cc:648] HybridClock initialized: now 1786375269101502 us; error 35 us; skew 500 ppm
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375269, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:09.104391 14137 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:21:09.104894 14137 webserver.cc:460] Webserver started at http://127.4.64.67:43857/ using document root <none> and password file <none>
I20260810 15:21:09.105187 14137 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:09.105262 14137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:09.105436 14137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:09.106627 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "4db075619e9f422b8f4b48d5d393db56"
format_stamp: "Formatted at 2026-08-10 15:21:09 on dist-test-slave-9zdj"
server_key: "229210f8e20ac6f746437afb8ab1d139"
server_key_iv: "3148f64becff7371b54e6b4faa35ac46"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.107024 14137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "4db075619e9f422b8f4b48d5d393db56"
format_stamp: "Formatted at 2026-08-10 15:21:09 on dist-test-slave-9zdj"
server_key: "229210f8e20ac6f746437afb8ab1d139"
server_key_iv: "3148f64becff7371b54e6b4faa35ac46"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.108497 14137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:21:09.109319 14155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.109588 14137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:09.109748 14137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "4db075619e9f422b8f4b48d5d393db56"
format_stamp: "Formatted at 2026-08-10 15:21:09 on dist-test-slave-9zdj"
server_key: "229210f8e20ac6f746437afb8ab1d139"
server_key_iv: "3148f64becff7371b54e6b4faa35ac46"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.109843 14137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:09.124169 14137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:09.126317 14137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:09.126495 14137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:09.126742 14137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:09.127079 14137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:09.127115 14137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.127158 14137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:09.127187 14137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.133416 14137 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:33251
I20260810 15:21:09.133482 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:33251 every 8 connection(s)
I20260810 15:21:09.133849 14137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:21:09.134547 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 14137
I20260810 15:21:09.134675 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:21:09.134900 4353 external_mini_cluster.cc:1428] Setting key 08b83ad2c820ecdd6c6950d1a09bfb13
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375269, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:09.144773 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.134702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:33579 (local address 127.4.64.126:34549)
0810 15:21:09.134758 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.134761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.135060 (+ 299us) server_negotiation.cc:407] Connection header received
0810 15:21:09.135905 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.135906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.135917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.135963 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.137120 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.137709 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.138778 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.138992 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.139078 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.141488 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:09.141490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:09.141490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:09.141499 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:09.143148 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:09.143627 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:09.143628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:09.143629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:09.143692 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:09.143974 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:09.143975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:09.143976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:09.144150 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:09.144249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:09.144581 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 15:21:09.144673 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:21:09.144982 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.134672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:33579)
0810 15:21:09.134899 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.134952 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.135799 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.136035 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.136039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.136814 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.137017 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.137022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.137815 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.137816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.138668 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.138671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.139090 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.139091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.139240 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.139818 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.139827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.141377 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:09.143264 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:09.143274 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:09.143277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:09.143543 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:09.143781 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:09.143782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:09.143783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:09.143906 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:09.144238 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:09.144247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:09.144434 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 15:21:09.144549 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:21:09.144733 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260810 15:21:09.145517 14296 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34549
I20260810 15:21:09.145674 14296 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:09.145900 14296 heartbeater.cc:511] Master 127.4.64.126:34549 requested a full tablet report, sending...
I20260810 15:21:09.146344 13745 ts_manager.cc:194] Registered new tserver with Master: 4db075619e9f422b8f4b48d5d393db56 (127.4.64.67:33251)
I20260810 15:21:09.146925 13745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:33579
I20260810 15:21:09.147176 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab.
Entry for principal oryx/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab.
WARNING: no policy specified for oryx/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab.
Entry for principal oryx/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab.
WARNING: no policy specified for oryx/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab.
Entry for principal oryx/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab.
I20260810 15:21:09.233597 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 13820
I20260810 15:21:09.246687 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 13972
I20260810 15:21:09.255301 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 14137
I20260810 15:21:09.263808 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 13707
I20260810 15:21:09.272877 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34549
--webserver_interface=127.4.64.126
--webserver_port=46705
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:09.359182 14305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:09.359480 14305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:09.359527 14305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:09.360797 14305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:09.360898 14305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:09.360921 14305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:09.360942 14305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:09.360971 14305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:09.362632 14305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=46705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:09.362982 14305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:09.363242 14305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:09.368706 14314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.368741 14312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.368891 14305 server_base.cc:1034] running on GCE node
W20260810 15:21:09.368990 14311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.369270 14305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:09.369591 14305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:09.370807 14305 hybrid_clock.cc:648] HybridClock initialized: now 1786375269370793 us; error 47 us; skew 500 ppm
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375269, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:09.373512 14305 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:21:09.374050 14305 webserver.cc:460] Webserver started at http://127.4.64.126:46705/ using document root <none> and password file <none>
I20260810 15:21:09.374279 14305 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:09.374336 14305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:09.375756 14305 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:09.376596 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.376823 14305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:09.376907 14305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "68573456fa6a4814b633a31019280214"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "2180c037c19584a5061af5e7771e29d9"
server_key_iv: "2f5246947a232321da16a6a4819d29e9"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.377249 14305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:09.411955 14305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:09.414551 14305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:09.414687 14305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:09.419412 14305 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34549
I20260810 15:21:09.419487 14382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34549 every 8 connection(s)
I20260810 15:21:09.419874 14305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:21:09.422165 14383 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:21:09.423035 14383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Bootstrap starting.
I20260810 15:21:09.429491 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 14305
I20260810 15:21:09.429925 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:42937
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:34549
--webserver_port=33311
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:21:09.432927 14383 log.cc:824] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:09.435081 14383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:21:09.435408 14383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Bootstrap complete.
I20260810 15:21:09.437927 14383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:09.438198 14383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68573456fa6a4814b633a31019280214, State: Initialized, Role: FOLLOWER
I20260810 15:21:09.438431 14383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:09.438535 14383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:21:09.438592 14383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:21:09.438642 14383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:21:09.439745 14383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:09.439879 14383 leader_election.cc:302] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68573456fa6a4814b633a31019280214; no voters:
I20260810 15:21:09.440101 14383 leader_election.cc:288] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:21:09.440143 14387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:21:09.440274 14387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [term 2 LEADER]: Becoming Leader. State: Replica: 68573456fa6a4814b633a31019280214, State: Running, Role: LEADER
I20260810 15:21:09.440390 14387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } }
I20260810 15:21:09.440420 14383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:21:09.442260 14389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68573456fa6a4814b633a31019280214. Latest consensus state: current_term: 2 leader_uuid: "68573456fa6a4814b633a31019280214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } } }
I20260810 15:21:09.442704 14389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:09.442447 14388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "68573456fa6a4814b633a31019280214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68573456fa6a4814b633a31019280214" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34549 } } }
I20260810 15:21:09.442909 14388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:09.443106 14405 catalog_manager.cc:1259] Loaded cluster ID: ceaf011109f84d9d811ae62e72421bbb
I20260810 15:21:09.443167 14405 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: loading cluster ID for follower catalog manager: success
I20260810 15:21:09.444447 14405 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: acquiring CA information for follower catalog manager: success
I20260810 15:21:09.444801 14405 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:21:09.445785 14408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:09.445992 14408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:09.446067 14408 catalog_manager.cc:1259] Loaded cluster ID: ceaf011109f84d9d811ae62e72421bbb
I20260810 15:21:09.446096 14408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:21:09.446269 14408 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:09.446342 14408 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 68573456fa6a4814b633a31019280214: Loaded TSK: 0
I20260810 15:21:09.446480 14408 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:21:09.518270 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:09.518486 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:09.518512 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:09.519855 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:09.519909 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:09.519937 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:21:09.521446 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.65.keytab
--principal=oryx/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:42937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=33311
--tserver_master_addrs=127.4.64.126:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:09.521724 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:09.522001 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:09.528417 14414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.528431 14416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.528431 14413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.528904 14385 server_base.cc:1034] running on GCE node
I20260810 15:21:09.529299 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:09.529646 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:09.530840 14385 hybrid_clock.cc:648] HybridClock initialized: now 1786375269530816 us; error 47 us; skew 500 ppm
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375269, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:09.534323 14385 init.cc:377] Logged in from keytab as oryx/127.4.64.65@KRBTEST.COM (short username oryx)
I20260810 15:21:09.534968 14385 webserver.cc:460] Webserver started at http://127.4.64.65:33311/ using document root <none> and password file <none>
I20260810 15:21:09.535318 14385 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:09.535398 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:09.537166 14385 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:09.537958 14425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.538183 14385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:09.538338 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "430e4ffad5f54a4b8c8d4b3b4875ac78"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "6d2a39720574249e2bd09d3df90e6624"
server_key_iv: "67eb39d716900893e61abdd5c8c86973"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.538779 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:09.551366 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:09.554145 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:09.554340 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:09.554662 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:09.555054 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:09.555117 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.555166 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:09.555197 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.563954 14385 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:42937
I20260810 15:21:09.564033 14559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:42937 every 8 connection(s)
I20260810 15:21:09.564479 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:21:09.567150 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 14385
I20260810 15:21:09.567525 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:34233
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:34549
--webserver_port=33389
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.576660 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.565720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:53227)
0810 15:21:09.565947 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.565999 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.566656 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.567152 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.567157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.568046 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.568268 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.568273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.569851 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.569852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.571284 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.571288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.572284 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.572285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.572431 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.573114 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.573124 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.576366 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
W20260810 15:21:09.576686 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.565756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:53227 (local address 127.4.64.126:34549)
0810 15:21:09.565974 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.565985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.566103 (+ 118us) server_negotiation.cc:407] Connection header received
0810 15:21:09.566880 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.566887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.566933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.567108 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.568410 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.569704 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.571394 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.572165 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.572307 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.576525 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:53227: BlockingRecv error: recv got EOF from 127.4.64.65:53227 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260810 15:21:09.577024 14560 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:34549 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.64.126:34549: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.583421 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.577327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:49773)
0810 15:21:09.577375 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.577387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.577427 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.577567 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.577568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.577794 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.577867 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.577870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.578765 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.578766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.579830 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.579834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.580452 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.580454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.580521 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.581155 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.581159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.583340 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 15:21:09.583586 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.577398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:49773 (local address 127.4.64.126:34549)
0810 15:21:09.577437 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.577438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.577450 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:21:09.577478 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.577479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.577490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.577546 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.577959 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.578655 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.579952 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.580329 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.580419 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.583506 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:49773: BlockingRecv error: recv got EOF from 127.4.64.65:49773 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.589391 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.583728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:43459)
0810 15:21:09.583938 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.583951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.584013 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.584164 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.584165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.584315 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.584369 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.584370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.585399 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.585400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.586062 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.586064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.586583 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.586584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.586635 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.587148 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.587153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.589282 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":123,"threads_started":1}
W20260810 15:21:09.589536 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.583959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:43459 (local address 127.4.64.126:34549)
0810 15:21:09.584023 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.584026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.584037 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:09.584068 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.584069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.584082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.584150 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.584442 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.585306 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.586141 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.586480 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.586548 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.589391 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:43459: BlockingRecv error: recv got EOF from 127.4.64.65:43459 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 15:21:09.589577 14560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:21:09.663903 14564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:09.664129 14564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:09.664160 14564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:09.665390 14564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:09.665437 14564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:09.665464 14564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:21:09.667025 14564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.66.keytab
--principal=oryx/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:34233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=33389
--tserver_master_addrs=127.4.64.126:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:09.667297 14564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:09.667547 14564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:09.673059 14571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.673262 14564 server_base.cc:1034] running on GCE node
W20260810 15:21:09.673064 14574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.673077 14570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.673633 14564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:09.673898 14564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:09.675071 14564 hybrid_clock.cc:648] HybridClock initialized: now 1786375269675054 us; error 45 us; skew 500 ppm
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375269, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:09.677896 14564 init.cc:377] Logged in from keytab as oryx/127.4.64.66@KRBTEST.COM (short username oryx)
I20260810 15:21:09.678436 14564 webserver.cc:460] Webserver started at http://127.4.64.66:33389/ using document root <none> and password file <none>
I20260810 15:21:09.678685 14564 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:09.678776 14564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:09.680542 14564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:09.681476 14583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.681737 14564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:09.681847 14564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "de914d9907c04cefac02fd6d013a3f95"
format_stamp: "Formatted at 2026-08-10 15:21:08 on dist-test-slave-9zdj"
server_key: "279619a0f4a0e2bc7ec074de03759120"
server_key_iv: "18aa1b19c308e1b5f893877fae47bbba"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.682250 14564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:09.694370 14564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:09.697271 14564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:09.697451 14564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:09.697783 14564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:09.698220 14564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:09.698267 14564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.698298 14564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:09.698316 14564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.704684 14564 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:34233
I20260810 15:21:09.704819 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:34233 every 8 connection(s)
I20260810 15:21:09.705173 14564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:21:09.713233 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.706038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:38969 (local address 127.4.64.126:34549)
0810 15:21:09.706181 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.706183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.706389 (+ 206us) server_negotiation.cc:407] Connection header received
0810 15:21:09.706959 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.706960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.706977 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.707032 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.708077 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.708658 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.709769 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.709968 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.710077 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.713174 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:38969: BlockingRecv error: recv got EOF from 127.4.64.66:38969 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 15:21:09.713295 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.706020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:38969)
0810 15:21:09.706246 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.706298 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.706847 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.707103 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.707108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.707720 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.707959 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.707964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.708780 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.708781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.709664 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.709667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.710086 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.710087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.710235 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.710780 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.710790 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.713081 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
W20260810 15:21:09.713593 14712 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:34549 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.64.126:34549: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
I20260810 15:21:09.714812 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 14564
I20260810 15:21:09.715103 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:33251
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:34549
--webserver_port=43857
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.718792 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.713926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:33229)
0810 15:21:09.713981 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.713992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.714039 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.714169 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.714170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.714321 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.714384 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.714385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.715106 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.715108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.715713 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.715715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.716115 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.716116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.716187 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.716678 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.716683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.718681 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 15:21:09.718827 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.713997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:33229 (local address 127.4.64.126:34549)
0810 15:21:09.714044 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.714046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.714054 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:09.714101 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.714102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.714112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.714162 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.714473 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.715019 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.715820 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.716028 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.716113 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.718773 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:33229: BlockingRecv error: recv got EOF from 127.4.64.66:33229 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.724112 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.719220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:39341)
0810 15:21:09.719266 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.719275 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.719309 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.719472 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.719473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.719626 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.719669 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.719671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.720355 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.720357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.721253 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.721255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.721581 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.721583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.721687 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.722079 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.722085 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.724023 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:21:09.724167 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.719293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:39341 (local address 127.4.64.126:34549)
0810 15:21:09.719351 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.719353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.719368 (+ 15us) server_negotiation.cc:407] Connection header received
0810 15:21:09.719388 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.719389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.719397 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.719447 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.719745 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.720269 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.721335 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.721509 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.721557 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.724101 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:39341: BlockingRecv error: recv got EOF from 127.4.64.66:39341 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260810 15:21:09.724370 14712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:21:09.803517 14715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:09.803819 14715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:09.803864 14715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:09.805066 14715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:09.805160 14715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:09.805207 14715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:21:09.806910 14715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/krb5kdc/oryx_127.4.64.67.keytab
--principal=oryx/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:33251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=43857
--tserver_master_addrs=127.4.64.126:34549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:09.807207 14715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:09.807459 14715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:09.812948 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.813050 14715 server_base.cc:1034] running on GCE node
W20260810 15:21:09.812943 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:09.812943 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:09.813587 14715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:09.813941 14715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:09.815110 14715 hybrid_clock.cc:648] HybridClock initialized: now 1786375269815089 us; error 41 us; skew 500 ppm
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375269, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:09.817816 14715 init.cc:377] Logged in from keytab as oryx/127.4.64.67@KRBTEST.COM (short username oryx)
I20260810 15:21:09.818356 14715 webserver.cc:460] Webserver started at http://127.4.64.67:43857/ using document root <none> and password file <none>
I20260810 15:21:09.818617 14715 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:09.818727 14715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:09.820154 14715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:09.820801 14731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.821012 14715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:09.821121 14715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "4db075619e9f422b8f4b48d5d393db56"
format_stamp: "Formatted at 2026-08-10 15:21:09 on dist-test-slave-9zdj"
server_key: "229210f8e20ac6f746437afb8ab1d139"
server_key_iv: "3148f64becff7371b54e6b4faa35ac46"
server_key_version: "encryptionkey@0"
I20260810 15:21:09.821496 14715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:09.836311 14715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:09.838917 14715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:09.839049 14715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:09.839267 14715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:09.839548 14715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:09.839579 14715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.839602 14715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:09.839615 14715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:09.845883 14715 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:33251
I20260810 15:21:09.845989 14861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:33251 every 8 connection(s)
I20260810 15:21:09.846352 14715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:21:09.852530 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 14715
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:21:09.854462 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.847146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:43685 (local address 127.4.64.126:34549)
0810 15:21:09.847213 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.847215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.847444 (+ 229us) server_negotiation.cc:407] Connection header received
0810 15:21:09.847983 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.847985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.847997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.848074 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.849121 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.849738 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.850766 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.850988 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.851071 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.854398 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:43685: BlockingRecv error: recv got EOF from 127.4.64.67:43685 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:21:09.854466 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.847132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:43685)
0810 15:21:09.847312 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.847354 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.847869 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.848147 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.848160 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.848795 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.849005 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.849010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.849835 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.849836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.850665 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.850667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.851078 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.851079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.851214 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.851698 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.851707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.854306 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
W20260810 15:21:09.854861 14862 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:34549 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.64.126:34549: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.860844 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.855116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:54127)
0810 15:21:09.855169 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.855179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.855217 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.855532 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.855533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.855685 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.855742 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.855743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.856624 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.856629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.857572 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.857574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.858169 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.858171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.858333 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.858738 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.858743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.860790 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 15:21:09.860970 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.855169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54127 (local address 127.4.64.126:34549)
0810 15:21:09.855204 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.855206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.855214 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:09.855267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.855267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.855274 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.855520 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.855829 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.856497 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.857729 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.858018 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.858084 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.860898 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:54127: BlockingRecv error: recv got EOF from 127.4.64.67:54127 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:21:09.861878 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.853107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.4.64.126:34549)
0810 15:21:09.853238 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.853239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.853248 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:21:09.853286 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.853286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.853296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.853347 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.854060 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.854688 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.855774 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.856098 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.856169 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.857725 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:09.857731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:09.857731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:09.857744 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:09.859591 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:09.860101 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:09.860106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:09.860106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:09.860247 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:09.860578 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:09.860579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:09.860579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:09.860884 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:09.861570 (+ 686us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:09.861689 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 15:21:09.861807 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":85,"threads_started":1}
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:09 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:09.865968 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:09.861133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:49171)
0810 15:21:09.861184 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:09.861196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:09.861236 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:09.861401 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:09.861403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:09.861546 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:09.861585 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.861587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.862507 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.862508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.863173 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:09.863175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.863567 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.863568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:09.863622 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.864165 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:09.864171 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:09.865915 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 15:21:09.866112 14563 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:09.861204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:49171 (local address 127.4.64.126:34549)
0810 15:21:09.861240 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:09.861242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:09.861250 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:09.861285 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:09.861285 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:09.861295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:09.861342 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:09.861726 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.862379 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.863267 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:09.863473 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:09.863540 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:09.866034 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:49171: BlockingRecv error: recv got EOF from 127.4.64.67:49171 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:21:09.866127 14862 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 15:21:10 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:10 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:10.595206 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:10.590081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:52871)
0810 15:21:10.590233 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:10.590252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:10.590308 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:10.590496 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:10.590498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:10.590699 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:10.590758 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:10.590760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.591481 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.591482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:10.592026 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:10.592028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.592364 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.592365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:10.592427 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:10.593000 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:10.593007 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:10.595143 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
W20260810 15:21:10.595431 14870 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:10.590144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:52871 (local address 127.4.64.126:34549)
0810 15:21:10.590305 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:10.590307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:10.590320 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:21:10.590351 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:10.590352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:10.590363 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:10.590434 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:10.590864 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.591390 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.592102 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.592290 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.592343 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:10.595358 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:52871: BlockingRecv error: recv got EOF from 127.4.64.65:52871 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
Aug 10 15:21:10 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:10 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:10.730027 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:10.724780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:46005)
0810 15:21:10.724962 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:10.724975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:10.725064 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:10.725235 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:10.725236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:10.725410 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:10.725509 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:10.725512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.726222 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.726223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:10.726818 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:10.726820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.727171 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.727172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:10.727228 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:10.727694 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:10.727699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:10.729954 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
W20260810 15:21:10.730165 14870 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:10.724822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:46005 (local address 127.4.64.126:34549)
0810 15:21:10.724895 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:10.724897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:10.725090 (+ 193us) server_negotiation.cc:407] Connection header received
0810 15:21:10.725143 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:10.725144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:10.725155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:10.725198 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:10.725577 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.726135 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.726898 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.727085 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.727137 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:10.730060 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:46005: BlockingRecv error: recv got EOF from 127.4.64.66:46005 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 15:21:10 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:10 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:10.872210 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:10.866581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:55757)
0810 15:21:10.866757 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:10.866772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:10.866852 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:10.867005 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:10.867006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:10.867254 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:10.867326 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:10.867328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.867998 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.867999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:10.869015 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:10.869019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.869396 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.869398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:10.869475 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:10.869985 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:10.869992 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:10.872097 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
W20260810 15:21:10.872251 14870 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:10.866784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:55757 (local address 127.4.64.126:34549)
0810 15:21:10.866832 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:10.866834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:10.866846 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:21:10.866920 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:10.866921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:10.866932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:10.866977 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:10.867412 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.867916 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.869110 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:10.869311 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:10.869367 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:10.872182 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:55757: BlockingRecv error: recv got EOF from 127.4.64.67:55757 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 15:21:11 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:11 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:11.601382 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:11.595874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:58189)
0810 15:21:11.596084 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:11.596100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:11.596184 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:11.596358 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:11.596358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:11.596554 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:11.596614 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:11.596632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.597406 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.597407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:11.598152 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:11.598172 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.598619 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.598620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:11.598708 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:11.599239 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:11.599245 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:11.601319 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
W20260810 15:21:11.601670 14874 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:11.596010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:58189 (local address 127.4.64.126:34549)
0810 15:21:11.596188 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:11.596191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:11.596232 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:21:11.596259 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:11.596260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:11.596270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:11.596326 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:11.596741 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.597290 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.598268 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.598547 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.598609 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:11.601518 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:58189: BlockingRecv error: recv got EOF from 127.4.64.65:58189 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
Aug 10 15:21:11 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:11 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:11.735462 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:11.730655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:51211)
0810 15:21:11.730793 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:11.730812 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:11.730874 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:11.731010 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:11.731010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:11.731195 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:11.731248 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:11.731250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.731976 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.731977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:11.732600 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:11.732603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.732991 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.732992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:11.733045 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:11.733497 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:11.733501 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:11.735386 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
W20260810 15:21:11.735585 14874 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:11.730714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:51211 (local address 127.4.64.126:34549)
0810 15:21:11.730793 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:11.730795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:11.730875 (+ 80us) server_negotiation.cc:407] Connection header received
0810 15:21:11.730933 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:11.730934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:11.730945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:11.730987 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:11.731346 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.731868 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.732708 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.732908 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.732968 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:11.735493 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:51211: BlockingRecv error: recv got EOF from 127.4.64.66:51211 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 15:21:11 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:11 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:11.878283 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:11.872785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:46137)
0810 15:21:11.872930 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:11.872948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:11.873000 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:11.873327 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:11.873328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:11.873523 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:11.873574 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:11.873576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.874523 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.874524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:11.875079 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:11.875081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.875561 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.875563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:11.875616 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:11.876107 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:11.876112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:11.878213 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260810 15:21:11.878429 14874 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:11.873000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:46137 (local address 127.4.64.126:34549)
0810 15:21:11.873096 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:11.873098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:11.873106 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:11.873208 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:11.873210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:11.873223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:11.873298 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:11.873789 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.874439 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.875240 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:11.875464 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:11.875525 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:11.878324 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:46137: BlockingRecv error: recv got EOF from 127.4.64.67:46137 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 15:21:12 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:12 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:12.606648 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:12.601957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:52749)
0810 15:21:12.602121 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:12.602135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:12.602207 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:12.602371 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:12.602372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:12.602552 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:12.602608 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:12.602610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.603287 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.603288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:12.603832 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:12.603834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.604172 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.604172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:12.604214 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:12.604682 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:12.604687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:12.606586 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260810 15:21:12.606791 14878 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:12.602011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:52749 (local address 127.4.64.126:34549)
0810 15:21:12.602157 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:12.602159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:12.602190 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:21:12.602263 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:12.602263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:12.602274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:12.602324 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:12.602681 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.603216 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.603916 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.604105 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.604155 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:12.606696 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:52749: BlockingRecv error: recv got EOF from 127.4.64.65:52749 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
Aug 10 15:21:12 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:12 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:12.741159 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:12.735995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:40357)
0810 15:21:12.736145 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:12.736156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:12.736209 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:12.736410 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:12.736411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:12.736587 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:12.736646 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:12.736647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.737342 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.737343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:12.737946 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:12.737949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.738273 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.738273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:12.738336 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:12.738824 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:12.738828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:12.741049 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
W20260810 15:21:12.741216 14878 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:12.736187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:40357 (local address 127.4.64.126:34549)
0810 15:21:12.736273 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:12.736275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:12.736284 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:21:12.736313 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:12.736313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:12.736325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:12.736392 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:12.736747 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.737261 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.738023 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.738207 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.738261 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:12.741144 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:40357: BlockingRecv error: recv got EOF from 127.4.64.66:40357 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 15:21:12 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:12 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:12.884016 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:12.878978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:47131)
0810 15:21:12.879236 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:12.879248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:12.879323 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:12.879482 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:12.879482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:12.879679 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:12.879733 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:12.879735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.880479 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.880480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:12.881031 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:12.881032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.881440 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.881441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:12.881498 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:12.882028 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:12.882033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:12.883899 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":80,"threads_started":1}
W20260810 15:21:12.884054 14878 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:12.879093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:47131 (local address 127.4.64.126:34549)
0810 15:21:12.879244 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:12.879246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:12.879306 (+ 60us) server_negotiation.cc:407] Connection header received
0810 15:21:12.879372 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:12.879373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:12.879384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:12.879471 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:12.879809 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.880397 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.881104 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:12.881333 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:12.881413 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:12.883984 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:47131: BlockingRecv error: recv got EOF from 127.4.64.67:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 10 15:21:13 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:13 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:13.612112 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:13.607217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:34113)
0810 15:21:13.607399 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:13.607418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:13.607472 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:13.607634 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:13.607635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:13.607818 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:13.607882 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:13.607884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.608563 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.608563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:13.609136 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:13.609138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.609542 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.609543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:13.609588 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:13.610167 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:13.610172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:13.612042 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
W20260810 15:21:13.612265 14886 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:13.607319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:34113 (local address 127.4.64.126:34549)
0810 15:21:13.607468 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:13.607470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:13.607482 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:21:13.607522 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:13.607523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:13.607534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:13.607602 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:13.607952 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.608488 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.609258 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.609464 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.609530 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:13.612170 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:34113: BlockingRecv error: recv got EOF from 127.4.64.65:34113 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
Aug 10 15:21:13 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:13 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:13.747270 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:13.741880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:44303)
0810 15:21:13.742071 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:13.742086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:13.742173 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:13.742318 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:13.742318 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:13.742540 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:13.742613 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:13.742622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.743430 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.743431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:13.744310 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:13.744314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.744698 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.744699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:13.744751 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:13.745244 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:13.745248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:13.747202 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
W20260810 15:21:13.747403 14886 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:13.741955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:44303 (local address 127.4.64.126:34549)
0810 15:21:13.742030 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:13.742033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:13.742153 (+ 120us) server_negotiation.cc:407] Connection header received
0810 15:21:13.742225 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:13.742226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:13.742238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:13.742290 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:13.742712 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.743317 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.744399 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.744596 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.744658 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:13.747312 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:44303: BlockingRecv error: recv got EOF from 127.4.64.66:44303 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 15:21:13 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:13 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:13.889556 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:13.884556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:41561)
0810 15:21:13.884706 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:13.884724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:13.884840 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:13.885001 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:13.885002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:13.885192 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:13.885245 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:13.885247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.886146 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.886148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:13.886832 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:13.886834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.887155 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.887156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:13.887199 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:13.887661 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:13.887666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:13.889495 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
W20260810 15:21:13.889700 14886 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:13.884622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:41561 (local address 127.4.64.126:34549)
0810 15:21:13.884695 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:13.884697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:13.884805 (+ 108us) server_negotiation.cc:407] Connection header received
0810 15:21:13.884909 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:13.884909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:13.884921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:13.884979 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:13.885305 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.886062 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.886908 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:13.887090 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:13.887147 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:13.889593 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:41561: BlockingRecv error: recv got EOF from 127.4.64.67:41561 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 15:21:14 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:14 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:14.623543 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:14.612781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:43811)
0810 15:21:14.612993 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:14.613005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:14.613070 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:14.613225 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:14.613226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:14.613407 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:14.613461 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:14.613463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.614257 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.614258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:14.614925 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:14.614928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.615316 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.615317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:14.615383 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:14.621395 (+ 6012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:14.621401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:14.623425 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":115,"threads_started":1}
W20260810 15:21:14.623605 14890 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:14.612910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:43811 (local address 127.4.64.126:34549)
0810 15:21:14.613056 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:14.613057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:14.613068 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:14.613135 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:14.613136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:14.613146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:14.613200 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:14.613542 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.614160 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.615022 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.615230 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.615304 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:14.623534 (+ 8230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:43811: BlockingRecv error: recv got EOF from 127.4.64.65:43811 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
Aug 10 15:21:14 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:14 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:14.752494 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:14.747875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:50821)
0810 15:21:14.748064 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:14.748075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:14.748134 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:14.748267 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:14.748268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:14.748447 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:14.748500 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:14.748502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.749172 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.749172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:14.749771 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:14.749773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.750096 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.750097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:14.750138 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:14.750659 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:14.750663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:14.752431 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
W20260810 15:21:14.752643 14890 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:14.748069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:50821 (local address 127.4.64.126:34549)
0810 15:21:14.748147 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:14.748149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:14.748158 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:21:14.748183 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:14.748184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:14.748194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:14.748245 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:14.748569 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.749096 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.749841 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.750029 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.750083 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:14.752551 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:50821: BlockingRecv error: recv got EOF from 127.4.64.66:50821 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 15:21:14 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:14 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:14.896715 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:14.891054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:45555)
0810 15:21:14.891218 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:14.891232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:14.891299 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:14.891941 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:14.891943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:14.892196 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:14.892266 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:14.892269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.892992 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.892994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:14.893553 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:14.893554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.894021 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.894022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:14.894087 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:14.894563 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:14.894568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:14.896622 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260810 15:21:14.896818 14890 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:14.891517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:45555 (local address 127.4.64.126:34549)
0810 15:21:14.891580 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:14.891583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:14.891593 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:21:14.891620 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:14.891620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:14.891634 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:14.891876 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:14.892358 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.892900 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.893719 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:14.893928 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:14.894019 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:14.896737 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:45555: BlockingRecv error: recv got EOF from 127.4.64.67:45555 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 15:21:15 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:15 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:15.629127 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:15.624051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:47577)
0810 15:21:15.624235 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:15.624248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:15.624312 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:15.624473 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:15.624474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:15.624668 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:15.624721 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:15.624722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.625517 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.625521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:15.626128 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:15.626130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.626518 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.626519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:15.626582 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:15.627123 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:15.627128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:15.629054 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
W20260810 15:21:15.629274 14894 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:15.624121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:47577 (local address 127.4.64.126:34549)
0810 15:21:15.624314 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:15.624316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:15.624325 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:21:15.624367 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:15.624368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:15.624380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:15.624467 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:15.624805 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.625426 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.626230 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.626438 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.626519 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:15.629155 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:47577: BlockingRecv error: recv got EOF from 127.4.64.65:47577 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
Aug 10 15:21:15 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:15 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:15.759058 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:15.753196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:32949)
0810 15:21:15.753458 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:15.753476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:15.753557 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:15.753806 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:15.753808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:15.754017 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:15.754098 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:15.754100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.754932 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.754933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:15.755582 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:15.755618 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.756177 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.756178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:15.756255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:15.756862 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:15.756867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:15.758988 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":125,"threads_started":1}
W20260810 15:21:15.759202 14894 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:15.753437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:32949 (local address 127.4.64.126:34549)
0810 15:21:15.753535 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:15.753537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:15.753551 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:21:15.753640 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:15.753641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:15.753654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:15.753749 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:15.754213 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.754826 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.755757 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.756092 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.756173 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:15.759116 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:32949: BlockingRecv error: recv got EOF from 127.4.64.66:32949 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 15:21:15 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:15 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:15.902917 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:15.897333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:33777)
0810 15:21:15.897556 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:15.897568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:15.897669 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:15.897851 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:15.897852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:15.898042 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:15.898095 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:15.898096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.898896 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.898897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:15.899441 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:15.899443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.899844 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.899845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:15.899911 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:15.900512 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:15.900517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:15.902856 (+ 2339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":150,"threads_started":1}
W20260810 15:21:15.903047 14894 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:15.897546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:33777 (local address 127.4.64.126:34549)
0810 15:21:15.897638 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:15.897639 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:15.897651 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:21:15.897755 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:15.897756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:15.897768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:15.897833 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:15.898218 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.898790 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.899550 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:15.899769 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:15.899833 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:15.902965 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:33777: BlockingRecv error: recv got EOF from 127.4.64.67:33777 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 15:21:16 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:16 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:16.635082 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:16.629727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:48885)
0810 15:21:16.629921 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:16.629933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:16.630001 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:16.630210 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:16.630211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:16.630417 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:16.630485 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:16.630487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.631249 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.631251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:16.631828 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:16.631830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.632245 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.632246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:16.632298 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:16.632800 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:16.632804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:16.634982 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
W20260810 15:21:16.635159 14899 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:16.629846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:48885 (local address 127.4.64.126:34549)
0810 15:21:16.630004 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:16.630006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:16.630039 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:21:16.630064 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:16.630064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:16.630075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:16.630126 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:16.630567 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.631125 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.631906 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.632100 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.632152 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:16.635083 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:48885: BlockingRecv error: recv got EOF from 127.4.64.65:48885 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
Aug 10 15:21:16 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:16 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:16.764806 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:16.759595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:43497)
0810 15:21:16.759807 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:16.759822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:16.759923 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:16.760057 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:16.760058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:16.760263 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:16.760316 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:16.760318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.761028 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.761030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:16.761575 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:16.761577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.762028 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.762028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:16.762096 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:16.762633 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:16.762645 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:16.764707 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":123,"threads_started":1}
W20260810 15:21:16.764879 14899 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:16.759757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:43497 (local address 127.4.64.126:34549)
0810 15:21:16.759846 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:16.759848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:16.759899 (+ 51us) server_negotiation.cc:407] Connection header received
0810 15:21:16.759983 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:16.759984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:16.759998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:16.760051 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:16.760407 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.760938 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.761720 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.761932 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.762024 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:16.764813 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:43497: BlockingRecv error: recv got EOF from 127.4.64.66:43497 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 15:21:16 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:16 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:16.908980 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:16.903691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:34527)
0810 15:21:16.903852 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:16.903866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:16.903970 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:16.904254 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:16.904255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:16.904447 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:16.904504 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:16.904506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.905271 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.905272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:16.905879 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:16.905881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.906248 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.906248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:16.906292 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:16.906759 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:16.906764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:16.908852 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260810 15:21:16.909020 14899 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:16.903762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:34527 (local address 127.4.64.126:34549)
0810 15:21:16.903818 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:16.903821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:16.903969 (+ 148us) server_negotiation.cc:407] Connection header received
0810 15:21:16.904041 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:16.904042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:16.904058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:16.904123 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:16.904585 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.905113 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.905983 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:16.906175 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:16.906235 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:16.908948 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:34527: BlockingRecv error: recv got EOF from 127.4.64.67:34527 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 10 15:21:17 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:17 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:17.641465 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:17.635768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:60677)
0810 15:21:17.635900 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:17.635914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:17.635978 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:17.636176 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:17.636177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:17.636380 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:17.636434 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:17.636435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.637323 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.637324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:17.637958 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:17.637961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.638423 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.638424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:17.638469 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:17.638962 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:17.638966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:17.641361 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
W20260810 15:21:17.641640 14906 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:17.635888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:60677 (local address 127.4.64.126:34549)
0810 15:21:17.636039 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:17.636041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:17.636050 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:21:17.636075 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:17.636075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:17.636085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:17.636157 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:17.636550 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.637227 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.638047 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.638344 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.638411 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:17.641539 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:60677: BlockingRecv error: recv got EOF from 127.4.64.65:60677 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
Aug 10 15:21:17 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:17 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:17.771207 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:17.765467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:55895)
0810 15:21:17.765690 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:17.765702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:17.765783 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:17.765913 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:17.765914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:17.766123 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:17.766239 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:17.766241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.767080 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.767081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:17.767775 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:17.767778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.768237 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.768238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:17.768291 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:17.768902 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:17.768907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:17.771091 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260810 15:21:17.771255 14906 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:17.765581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:55895 (local address 127.4.64.126:34549)
0810 15:21:17.765693 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:17.765695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:17.765780 (+ 85us) server_negotiation.cc:407] Connection header received
0810 15:21:17.765847 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:17.765848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:17.765864 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:17.765905 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:17.766311 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.766981 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.767892 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.768094 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.768154 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:17.771204 (+ 3050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:55895: BlockingRecv error: recv got EOF from 127.4.64.66:55895 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 15:21:17 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:17 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:17.914618 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:17.909574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:49647)
0810 15:21:17.909774 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:17.909788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:17.909854 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:17.910023 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:17.910024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:17.910319 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:17.910376 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:17.910379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.911165 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.911166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:17.911718 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:17.911720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.912095 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.912096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:17.912147 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:17.912638 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:17.912643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:17.914554 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":138,"threads_started":1}
W20260810 15:21:17.914816 14906 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:17.909753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:49647 (local address 127.4.64.126:34549)
0810 15:21:17.909830 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:17.909832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:17.909843 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:17.909916 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:17.909916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:17.909935 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:17.910002 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:17.910470 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.911047 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.911809 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:17.912018 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:17.912079 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:17.914721 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:49647: BlockingRecv error: recv got EOF from 127.4.64.67:49647 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 15:21:18 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:18 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:18.647148 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:18.642101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:51505)
0810 15:21:18.642274 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:18.642287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:18.642384 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:18.642533 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:18.642533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:18.642710 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:18.642766 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:18.642768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.643563 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.643564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:18.644103 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:18.644105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.644452 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.644453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:18.644493 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:18.644940 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:18.644944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:18.647086 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
W20260810 15:21:18.647354 14911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:18.642157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:51505 (local address 127.4.64.126:34549)
0810 15:21:18.642320 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:18.642321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:18.642346 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:21:18.642442 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:18.642443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:18.642453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:18.642515 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:18.642852 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.643491 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.644174 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.644383 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.644439 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:18.647264 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:51505: BlockingRecv error: recv got EOF from 127.4.64.65:51505 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
Aug 10 15:21:18 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:18 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:18.777024 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:18.771769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:58659)
0810 15:21:18.771944 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:18.771961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:18.772019 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:18.772161 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:18.772162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:18.772354 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:18.772416 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:18.772418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.773153 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.773153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:18.773761 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:18.773764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.774111 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.774112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:18.774159 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:18.774650 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:18.774655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:18.776896 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
W20260810 15:21:18.777064 14911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:18.771891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:58659 (local address 127.4.64.126:34549)
0810 15:21:18.771965 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:18.771967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:18.772011 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:21:18.772071 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:18.772071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:18.772081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:18.772138 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:18.772518 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.773058 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.773840 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.774037 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.774095 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:18.776996 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:58659: BlockingRecv error: recv got EOF from 127.4.64.66:58659 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 15:21:18 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:18 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:18.920568 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:18.915224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:52181)
0810 15:21:18.915492 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:18.915506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:18.915585 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:18.915734 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:18.915735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:18.915927 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:18.915984 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:18.915986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.916744 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.916745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:18.917387 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:18.917389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.917859 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.917861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:18.917931 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:18.918415 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:18.918420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:18.920449 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":195,"threads_started":1}
W20260810 15:21:18.920606 14911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:18.915317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:52181 (local address 127.4.64.126:34549)
0810 15:21:18.915367 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:18.915369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:18.915580 (+ 211us) server_negotiation.cc:407] Connection header received
0810 15:21:18.915633 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:18.915634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:18.915645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:18.915699 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:18.916084 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.916639 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.917476 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:18.917687 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:18.917808 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:18.920538 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:52181: BlockingRecv error: recv got EOF from 127.4.64.67:52181 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 10 15:21:19 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:19 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:19.656669 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:19.647653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:60457)
0810 15:21:19.647831 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:19.647844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:19.647897 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:19.648120 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:19.648121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:19.648323 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:19.648382 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:19.648384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.649200 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.649201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:19.649826 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:19.649828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.650236 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.650237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:19.650285 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:19.654343 (+ 4058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:19.654349 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:19.656557 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
W20260810 15:21:19.656757 14915 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:19.647789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:60457 (local address 127.4.64.126:34549)
0810 15:21:19.647948 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:19.647950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:19.647958 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:21:19.647982 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:19.647983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:19.647995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:19.648068 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:19.648486 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.649097 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.649927 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.650156 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.650225 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:19.656665 (+ 6440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:60457: BlockingRecv error: recv got EOF from 127.4.64.65:60457 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
Aug 10 15:21:19 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:19 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:19.783030 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:19.777829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:52957)
0810 15:21:19.778021 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:19.778034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:19.778134 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:19.778272 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:19.778273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:19.778469 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:19.778522 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:19.778524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.779567 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.779568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:19.780118 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:19.780120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.780548 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.780549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:19.780617 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:19.781067 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:19.781071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:19.782968 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
W20260810 15:21:19.783166 14915 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:19.777943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:52957 (local address 127.4.64.126:34549)
0810 15:21:19.778012 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:19.778014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:19.778110 (+ 96us) server_negotiation.cc:407] Connection header received
0810 15:21:19.778198 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:19.778199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:19.778211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:19.778267 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:19.778611 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.779471 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.780222 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.780441 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.780525 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:19.783080 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:52957: BlockingRecv error: recv got EOF from 127.4.64.66:52957 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 10 15:21:19 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:19 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:19.926745 14918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:19.921233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:34167)
0810 15:21:19.921406 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:19.921429 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:19.921499 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:19.921695 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:19.921696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:19.921887 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:19.921939 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:19.921941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.922975 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.922976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:19.923581 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:19.923583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.924002 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.924003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:19.924063 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:19.924681 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:19.924686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:19.926674 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
W20260810 15:21:19.926932 14915 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:19.921460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:34167 (local address 127.4.64.126:34549)
0810 15:21:19.921533 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:19.921535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:19.921546 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:19.921581 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:19.921582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:19.921600 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:19.921674 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:19.922064 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.922885 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.923662 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:19.923869 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:19.923987 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:19.926860 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:34167: BlockingRecv error: recv got EOF from 127.4.64.67:34167 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 15:21:20 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:20 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:20.662670 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:20.657398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:33525)
0810 15:21:20.657590 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:20.657602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:20.657726 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:20.657891 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:20.657892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:20.658080 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:20.658135 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:20.658137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.658865 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.658866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:20.659405 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:20.659407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.659784 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.659786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:20.659845 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:20.660377 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:20.660383 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:20.662542 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":116,"threads_started":1}
W20260810 15:21:20.662753 14920 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:20.657529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:33525 (local address 127.4.64.126:34549)
0810 15:21:20.657671 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:20.657673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:20.657710 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:21:20.657792 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:20.657792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:20.657805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:20.657870 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:20.658200 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.658777 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.659488 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.659684 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.659744 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:20.662667 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:33525: BlockingRecv error: recv got EOF from 127.4.64.65:33525 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":110,"threads_started":1}
Aug 10 15:21:20 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:20 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:20.789842 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:20.783668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:34355)
0810 15:21:20.783949 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:20.783969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:20.784067 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:20.784290 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:20.784291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:20.784625 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:20.784681 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:20.784683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.785662 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.785664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:20.786329 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:20.786332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.786737 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.786738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:20.786802 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:20.787338 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:20.787343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:20.789739 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":115,"threads_started":1}
W20260810 15:21:20.789963 14920 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:20.783786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:34355 (local address 127.4.64.126:34549)
0810 15:21:20.783919 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:20.783922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:20.784043 (+ 121us) server_negotiation.cc:407] Connection header received
0810 15:21:20.784186 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:20.784187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:20.784202 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:20.784283 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:20.784792 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.785544 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.786421 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.786642 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.786735 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:20.789895 (+ 3160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:34355: BlockingRecv error: recv got EOF from 127.4.64.66:34355 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 15:21:20 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:20 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:20.932770 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:20.927312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:50661)
0810 15:21:20.927496 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:20.927507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:20.927573 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:20.927732 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:20.927733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:20.927941 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:20.927994 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:20.927996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.928751 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.928753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:20.929403 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:20.929406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.929892 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.929893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:20.929983 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:20.930750 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:20.930757 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:20.932684 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
W20260810 15:21:20.932924 14920 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:20.927426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:50661 (local address 127.4.64.126:34549)
0810 15:21:20.927484 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:20.927486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:20.927559 (+ 73us) server_negotiation.cc:407] Connection header received
0810 15:21:20.927627 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:20.927628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:20.927642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:20.927707 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:20.928069 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.928631 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.929528 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:20.929758 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:20.929874 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:20.932814 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:50661: BlockingRecv error: recv got EOF from 127.4.64.67:50661 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 15:21:21 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:21 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:21.669462 14924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:21.663292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:47767)
0810 15:21:21.663507 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:21.663521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:21.663598 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:21.663856 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:21.663858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:21.664109 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:21.664165 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:21.664168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.664928 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.664929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:21.665577 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:21.665579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.666156 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.666158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:21.666274 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:21.666895 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:21.666901 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:21.669384 (+ 2483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
W20260810 15:21:21.669714 14925 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:21.663517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:47767 (local address 127.4.64.126:34549)
0810 15:21:21.663651 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:21.663652 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:21.663662 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:21:21.663692 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:21.663692 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:21.663704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:21.663769 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:21.664260 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.664815 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.665817 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.666079 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.666139 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:21.669587 (+ 3448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:47767: BlockingRecv error: recv got EOF from 127.4.64.65:47767 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
Aug 10 15:21:21 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:21 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:21.795734 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:21.790581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:49217)
0810 15:21:21.790735 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:21.790752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:21.790812 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:21.790965 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:21.790966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:21.791153 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:21.791207 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:21.791209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.792075 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.792076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:21.792639 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:21.792640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.793065 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.793065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:21.793116 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:21.793683 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:21.793690 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:21.795670 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260810 15:21:21.795955 14925 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:21.790644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:49217 (local address 127.4.64.126:34549)
0810 15:21:21.790747 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:21.790750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:21.790843 (+ 93us) server_negotiation.cc:407] Connection header received
0810 15:21:21.790872 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:21.790873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:21.790886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:21.790944 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:21.791354 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.791978 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.792732 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.792987 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.793052 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:21.795868 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:49217: BlockingRecv error: recv got EOF from 127.4.64.66:49217 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 15:21:21 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:21 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:21.938760 14927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:21.933382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:54611)
0810 15:21:21.933567 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:21.933586 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:21.933662 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:21.933858 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:21.933859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:21.934043 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:21.934095 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:21.934096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.934794 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.934795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:21.935464 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:21.935467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.935874 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.935875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:21.935932 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:21.936448 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:21.936459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:21.938641 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
W20260810 15:21:21.938798 14925 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:21.933509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54611 (local address 127.4.64.126:34549)
0810 15:21:21.933646 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:21.933648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:21.933663 (+ 15us) server_negotiation.cc:407] Connection header received
0810 15:21:21.933745 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:21.933746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:21.933758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:21.933835 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:21.934167 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.934675 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.935567 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:21.935761 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:21.935840 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:21.938731 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:54611: BlockingRecv error: recv got EOF from 127.4.64.67:54611 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 15:21:22 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:22 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:22.675045 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:22.670107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:49009)
0810 15:21:22.670280 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:22.670298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:22.670363 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:22.670515 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:22.670515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:22.670700 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:22.670754 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:22.670756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.671450 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.671451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:22.671996 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:22.671997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.672378 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.672378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:22.672428 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:22.672896 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:22.672901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:22.674925 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
W20260810 15:21:22.675101 14930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:22.670159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:49009 (local address 127.4.64.126:34549)
0810 15:21:22.670310 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:22.670311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:22.670347 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:21:22.670429 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:22.670429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:22.670442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:22.670494 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:22.670829 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.671363 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.672077 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.672287 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.672348 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:22.675024 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:49009: BlockingRecv error: recv got EOF from 127.4.64.65:49009 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
Aug 10 15:21:22 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:22 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:22.801781 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:22.796352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:56915)
0810 15:21:22.796543 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:22.796558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:22.796633 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:22.796825 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:22.796826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:22.797026 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:22.797082 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:22.797084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.798129 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.798131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:22.798696 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:22.798698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.799084 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.799085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:22.799133 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:22.799624 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:22.799628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:22.801676 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
W20260810 15:21:22.801970 14930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:22.796580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:56915 (local address 127.4.64.126:34549)
0810 15:21:22.796677 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:22.796678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:22.796705 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:21:22.796736 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:22.796737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:22.796748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:22.796792 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:22.797265 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.798020 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.798796 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.799002 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.799065 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:22.801876 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:56915: BlockingRecv error: recv got EOF from 127.4.64.66:56915 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 15:21:22 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:22 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:22.945274 14932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:22.939405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:40319)
0810 15:21:22.939586 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:22.939606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:22.939709 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:22.939944 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:22.939945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:22.940155 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:22.940209 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:22.940211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.941136 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.941137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:22.942063 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:22.942067 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.942550 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.942551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:22.942620 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:22.943202 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:22.943208 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:22.945196 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
W20260810 15:21:22.945516 14930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:22.939595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:40319 (local address 127.4.64.126:34549)
0810 15:21:22.939673 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:22.939675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:22.939686 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:22.939832 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:22.939833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:22.939847 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:22.939919 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:22.940301 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.941019 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.942172 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:22.942453 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:22.942538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:22.945447 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:40319: BlockingRecv error: recv got EOF from 127.4.64.67:40319 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 15:21:23 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:23 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:23.680964 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:23.675653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:53239)
0810 15:21:23.675870 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:23.675883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:23.675948 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:23.676100 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:23.676101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:23.676293 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:23.676346 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:23.676347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.677126 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.677127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:23.677710 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:23.677713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.678084 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.678085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:23.678131 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:23.678699 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:23.678704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:23.680869 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":109,"threads_started":1}
W20260810 15:21:23.681099 14934 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:23.675784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:53239 (local address 127.4.64.126:34549)
0810 15:21:23.675933 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:23.675935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:23.675946 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:23.676002 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:23.676003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:23.676014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:23.676077 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:23.676508 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.677049 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.677802 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.678008 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.678070 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:23.681002 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:53239: BlockingRecv error: recv got EOF from 127.4.64.65:53239 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
Aug 10 15:21:23 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:23 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:23.807785 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:23.802456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:47387)
0810 15:21:23.802628 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:23.802639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:23.802713 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:23.802854 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:23.802855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:23.803043 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:23.803094 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:23.803095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.803944 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.803945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:23.804704 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:23.804707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.805098 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.805099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:23.805159 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:23.805658 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:23.805664 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:23.807721 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
W20260810 15:21:23.807955 14934 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:23.802578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:47387 (local address 127.4.64.126:34549)
0810 15:21:23.802627 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:23.802629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:23.802695 (+ 66us) server_negotiation.cc:407] Connection header received
0810 15:21:23.802763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:23.802764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:23.802776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:23.802824 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:23.803276 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.803862 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.804816 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.805020 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.805096 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:23.807860 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:47387: BlockingRecv error: recv got EOF from 127.4.64.66:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 15:21:23 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:23 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.67@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:23.950973 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:23.945817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.67:34565)
0810 15:21:23.946000 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:23.946014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:23.946091 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:23.946258 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:23.946259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:23.946457 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:23.946511 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:23.946513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.947218 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.947219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:23.947867 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:23.947869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.948234 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.948235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:23.948291 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:23.948760 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:23.948765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:23.950908 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
W20260810 15:21:23.951217 14934 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:23.946012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:34565 (local address 127.4.64.126:34549)
0810 15:21:23.946084 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:23.946085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:23.946095 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:21:23.946152 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:23.946152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:23.946165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:23.946215 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:23.946599 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.947117 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.947951 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:23.948141 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:23.948197 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:23.951129 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.67:34565: BlockingRecv error: recv got EOF from 127.4.64.67:34565 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 15:21:24 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:24 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.65@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:21:24.690404 14939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:24.681603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.65:60123)
0810 15:21:24.681887 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:24.681901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:24.681973 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:24.682123 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:24.682123 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:24.682303 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:24.682398 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:24.682400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.683070 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.683071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:24.683640 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:24.683643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.684153 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.684155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:24.684214 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:24.688190 (+ 3976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:24.688197 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:24.690339 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
W20260810 15:21:24.690603 14937 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:24.681701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:60123 (local address 127.4.64.126:34549)
0810 15:21:24.681853 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:24.681855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:24.681970 (+ 115us) server_negotiation.cc:407] Connection header received
0810 15:21:24.682021 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:24.682022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:24.682032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:24.682103 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:24.682467 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.682992 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.683724 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.683922 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.683976 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:24.690498 (+ 6522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.65:60123: BlockingRecv error: recv got EOF from 127.4.64.65:60123 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
Aug 10 15:21:24 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:21:24 dist-test-slave-9zdj krb5kdc[13691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.64.66@KRBTEST.COM for oryx/127.4.64.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:21:24.814162 14937 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:24.808549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:35437 (local address 127.4.64.126:34549)
0810 15:21:24.808661 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:24.808663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:24.808697 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:21:24.808825 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:24.808826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:24.808843 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:24.808899 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:24.809353 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.810037 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.810874 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.811103 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.811191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:24.814097 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.64.66:35437: BlockingRecv error: recv got EOF from 127.4.64.66:35437 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 15:21:24.814090 14940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:24.808408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34549 (local address 127.4.64.66:35437)
0810 15:21:24.808615 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:24.808633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:24.808723 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:24.808919 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:24.808920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:24.809139 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:24.809247 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:24.809250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.810137 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.810138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:24.810746 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:24.810748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:24.811209 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:24.811210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:24.811285 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:24.811743 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:24.811748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:24.813981 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.64.126:34549: Server oryx/127.4.64.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260810 15:21:24.863214 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 14385
I20260810 15:21:24.871460 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 14564
I20260810 15:21:24.879309 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 14715
I20260810 15:21:24.887287 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 14305
2026-08-10T15:21:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17724 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T15:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:24Z Disabled control of system clock
I20260810 15:21:24.921353 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:45667
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:45667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 15:21:25.016742 14944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.016974 14944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.016999 14944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.018407 14944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:25.018462 14944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.018476 14944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.018491 14944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:25.018504 14944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:25.020072 14944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:45667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:45667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.020375 14944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.020776 14944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:21:25.027051 14944 server_base.cc:1034] running on GCE node
W20260810 15:21:25.027134 14953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.027019 14952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.026990 14957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.027518 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.027904 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.029065 14944 hybrid_clock.cc:648] HybridClock initialized: now 1786375285029028 us; error 56 us; skew 500 ppm
I20260810 15:21:25.030508 14944 webserver.cc:460] Webserver started at http://127.4.64.126:34727/ using document root <none> and password file <none>
I20260810 15:21:25.030789 14944 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.030902 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.031064 14944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.032184 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "bec298a7302d4db8bc1c0abf60df461c"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "388e6739a2cf30a075769527d4145617"
server_key_iv: "08af93362bba916be76884815a5401e3"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.032575 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "bec298a7302d4db8bc1c0abf60df461c"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "388e6739a2cf30a075769527d4145617"
server_key_iv: "08af93362bba916be76884815a5401e3"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.034147 14944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:25.034994 14963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.035336 14944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:21:25.035449 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "bec298a7302d4db8bc1c0abf60df461c"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "388e6739a2cf30a075769527d4145617"
server_key_iv: "08af93362bba916be76884815a5401e3"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.035527 14944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.043998 14944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.044330 14944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.044485 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.049110 14944 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:45667
I20260810 15:21:25.049145 15024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:45667 every 8 connection(s)
I20260810 15:21:25.049532 14944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:21:25.050453 15027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:25.054836 15027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c: Bootstrap starting.
I20260810 15:21:25.055505 15027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:25.055855 15027 log.cc:824] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:25.056629 15027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c: No bootstrap required, opened a new log
I20260810 15:21:25.057273 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 14944
I20260810 15:21:25.057375 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:21:25.057588 4353 external_mini_cluster.cc:1428] Setting key 12a44d1388e51a8a5f5cbf0dfe3e7c3d
I20260810 15:21:25.059273 15027 raft_consensus.cc:348] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec298a7302d4db8bc1c0abf60df461c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45667 } }
I20260810 15:21:25.059443 15027 raft_consensus.cc:374] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:25.059489 15027 raft_consensus.cc:729] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec298a7302d4db8bc1c0abf60df461c, State: Initialized, Role: FOLLOWER
I20260810 15:21:25.059635 15027 consensus_queue.cc:260] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec298a7302d4db8bc1c0abf60df461c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45667 } }
I20260810 15:21:25.059741 15027 raft_consensus.cc:388] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:21:25.059777 15027 raft_consensus.cc:482] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:21:25.059831 15027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:25.060391 15027 raft_consensus.cc:504] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec298a7302d4db8bc1c0abf60df461c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45667 } }
I20260810 15:21:25.060532 15027 leader_election.cc:302] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bec298a7302d4db8bc1c0abf60df461c; no voters:
I20260810 15:21:25.060724 15027 leader_election.cc:288] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:21:25.060766 15033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:25.060896 15033 raft_consensus.cc:686] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [term 1 LEADER]: Becoming Leader. State: Replica: bec298a7302d4db8bc1c0abf60df461c, State: Running, Role: LEADER
I20260810 15:21:25.060992 15027 sys_catalog.cc:564] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:21:25.060977 15033 consensus_queue.cc:237] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec298a7302d4db8bc1c0abf60df461c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45667 } }
I20260810 15:21:25.061326 15033 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bec298a7302d4db8bc1c0abf60df461c. Latest consensus state: current_term: 1 leader_uuid: "bec298a7302d4db8bc1c0abf60df461c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec298a7302d4db8bc1c0abf60df461c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45667 } } }
I20260810 15:21:25.061444 15033 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:25.061335 15035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bec298a7302d4db8bc1c0abf60df461c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec298a7302d4db8bc1c0abf60df461c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 45667 } } }
I20260810 15:21:25.061604 15035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:25.061868 15040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:25.062439 15040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:25.063921 15053 mvcc.cc:205] Tried to move back new op lower bound from 7316993167619477504 to 7316993167610060800. Current Snapshot: MvccSnapshot[applied={T|T < 7316993167619477504}]
I20260810 15:21:25.064970 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.058226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.4.64.126:45667)
0810 15:21:25.058561 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.058575 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.058602 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:21:25.059270 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.059278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.059362 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.059542 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.059937 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.061717 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.062868 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.063511 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.063628 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.064308 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:25.064312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:25.064313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:25.064483 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:25.064519 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:25.064531 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:25.064589 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:25.064739 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 15:21:25.064807 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":213,"threads_started":1}
I20260810 15:21:25.066098 15040 catalog_manager.cc:1347] Generated new cluster ID: a8f40dc1a3ff4a088d8108f90f85d3d6
I20260810 15:21:25.066164 15040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:21:25.079315 15040 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:21:25.079905 15040 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:25.083088 15040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bec298a7302d4db8bc1c0abf60df461c: Generated new TSK 0
I20260810 15:21:25.083290 15040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:21:25.090245 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45667
--builtin_ntp_servers=127.4.64.84:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 15:21:25.183743 15055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.184018 15055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.184055 15055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.185281 15055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.185357 15055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.185400 15055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:21:25.186923 15055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.187235 15055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.187588 15055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:25.193130 15065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.193112 15063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.193116 15062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.193411 15055 server_base.cc:1034] running on GCE node
I20260810 15:21:25.193684 15055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.193982 15055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.195159 15055 hybrid_clock.cc:648] HybridClock initialized: now 1786375285195138 us; error 35 us; skew 500 ppm
I20260810 15:21:25.196893 15055 webserver.cc:460] Webserver started at http://127.4.64.65:45519/ using document root <none> and password file <none>
I20260810 15:21:25.197170 15055 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.197230 15055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.197358 15055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.198906 15055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "7352d38135874a8aab28a85f57a76a47"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "93051aed94e4fc596abdc5036902dc13"
server_key_iv: "0d449a148eebef363c431d7b058c210a"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.199430 15055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "7352d38135874a8aab28a85f57a76a47"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "93051aed94e4fc596abdc5036902dc13"
server_key_iv: "0d449a148eebef363c431d7b058c210a"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.201412 15055 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260810 15:21:25.202409 15072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.202605 15055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:25.202715 15055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "7352d38135874a8aab28a85f57a76a47"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "93051aed94e4fc596abdc5036902dc13"
server_key_iv: "0d449a148eebef363c431d7b058c210a"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.202792 15055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.250787 15055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.251134 15055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.251259 15055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.251508 15055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:25.251843 15055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:25.251878 15055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.251904 15055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:25.251919 15055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.258948 15055 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:43513
I20260810 15:21:25.259003 15201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:43513 every 8 connection(s)
I20260810 15:21:25.259414 15055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:21:25.265470 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.260410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:49861 (local address 127.4.64.126:45667)
0810 15:21:25.260544 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.260547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.260779 (+ 232us) server_negotiation.cc:407] Connection header received
0810 15:21:25.261449 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.261450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.261463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.261525 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.262359 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.262893 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.263957 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.264189 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.264274 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.265132 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:25.265133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:25.265133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:25.265138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:25.265155 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:25.265167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:25.265238 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:25.265348 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 15:21:25.265383 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 15:21:25.265797 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.260307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45667 (local address 127.4.64.65:49861)
0810 15:21:25.260607 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:25.260645 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:25.261331 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:25.261638 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:25.261641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:25.261651 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:25.262262 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:25.262268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.262999 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.263000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:25.263852 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:25.263854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.264283 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.264284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:25.264429 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.264970 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:21:25.264970 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:25.265001 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:21:25.265024 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:25.265238 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:25.265246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:25.265249 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:25.265335 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 15:21:25.265449 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
I20260810 15:21:25.266410 15202 heartbeater.cc:347] Connected to a master server at 127.4.64.126:45667
I20260810 15:21:25.266503 15202 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:25.266714 15202 heartbeater.cc:511] Master 127.4.64.126:45667 requested a full tablet report, sending...
I20260810 15:21:25.267266 14984 ts_manager.cc:194] Registered new tserver with Master: 7352d38135874a8aab28a85f57a76a47 (127.4.64.65:43513)
I20260810 15:21:25.267989 14984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.65:49861
I20260810 15:21:25.268437 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15055
I20260810 15:21:25.268546 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:21:25.268761 4353 external_mini_cluster.cc:1428] Setting key b92f30c7beced6734097ef294328f639
I20260810 15:21:25.269883 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45667
--builtin_ntp_servers=127.4.64.84:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 15:21:25.354321 15205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.354545 15205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.354566 15205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.355732 15205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.355778 15205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.355803 15205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:21:25.357164 15205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.357445 15205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.357795 15205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:25.363396 15213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.363408 15215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.363440 15205 server_base.cc:1034] running on GCE node
W20260810 15:21:25.363400 15212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.363867 15205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.364156 15205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.365348 15205 hybrid_clock.cc:648] HybridClock initialized: now 1786375285365326 us; error 33 us; skew 500 ppm
I20260810 15:21:25.366672 15205 webserver.cc:460] Webserver started at http://127.4.64.66:44325/ using document root <none> and password file <none>
I20260810 15:21:25.366976 15205 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.367053 15205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.367213 15205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.368322 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "4f91835f913f4a1abfe1019df357eb0d"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "093e71a3d7a955070a42919f4388050a"
server_key_iv: "112a1530ba2d3a3ca6f9140c3c0f97e4"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.368697 15205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "4f91835f913f4a1abfe1019df357eb0d"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "093e71a3d7a955070a42919f4388050a"
server_key_iv: "112a1530ba2d3a3ca6f9140c3c0f97e4"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.370164 15205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:25.370994 15221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.371181 15205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:25.371294 15205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "4f91835f913f4a1abfe1019df357eb0d"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "093e71a3d7a955070a42919f4388050a"
server_key_iv: "112a1530ba2d3a3ca6f9140c3c0f97e4"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.371376 15205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.382433 15205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.382786 15205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.382948 15205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.383194 15205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:25.383550 15205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:25.383606 15205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.383651 15205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:25.383678 15205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.390036 15205 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:33187
I20260810 15:21:25.390093 15350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:33187 every 8 connection(s)
I20260810 15:21:25.390503 15205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:21:25.395834 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.391275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:51227 (local address 127.4.64.126:45667)
0810 15:21:25.391406 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.391408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.391593 (+ 185us) server_negotiation.cc:407] Connection header received
0810 15:21:25.392151 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.392152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.392168 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.392219 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.392890 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.393391 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.394471 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.394664 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.394721 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.395520 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:25.395521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:25.395521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:25.395525 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:25.395540 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:25.395544 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:25.395592 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:25.395742 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 15:21:25.395771 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 15:21:25.395988 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15205
I20260810 15:21:25.396098 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:21:25.396196 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.391248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45667 (local address 127.4.64.66:51227)
0810 15:21:25.391475 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:25.391510 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:25.392059 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:25.392281 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:25.392284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:25.392293 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:25.392807 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:25.392812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.393474 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.393474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:25.394393 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:25.394396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.394745 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.394746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:25.394881 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.395382 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:21:25.395383 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:25.395412 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:21:25.395434 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:25.395646 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:25.395654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:25.395657 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:25.395789 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 15:21:25.395920 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":114,"threads_started":1}
I20260810 15:21:25.396481 4353 external_mini_cluster.cc:1428] Setting key 23145b89fd837f2d2068bbb569a22f20
I20260810 15:21:25.396835 15351 heartbeater.cc:347] Connected to a master server at 127.4.64.126:45667
I20260810 15:21:25.396965 15351 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:25.397197 15351 heartbeater.cc:511] Master 127.4.64.126:45667 requested a full tablet report, sending...
I20260810 15:21:25.397593 14984 ts_manager.cc:194] Registered new tserver with Master: 4f91835f913f4a1abfe1019df357eb0d (127.4.64.66:33187)
I20260810 15:21:25.397609 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45667
--builtin_ntp_servers=127.4.64.84:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 15:21:25.398046 14984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.66:51227
W20260810 15:21:25.483675 15354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.483883 15354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.483904 15354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.484992 15354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.485025 15354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.485049 15354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:21:25.486390 15354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:45667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.486630 15354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.486893 15354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:25.492483 15363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.492503 15360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.492520 15354 server_base.cc:1034] running on GCE node
W20260810 15:21:25.492466 15361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.493003 15354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.493299 15354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.494446 15354 hybrid_clock.cc:648] HybridClock initialized: now 1786375285494418 us; error 41 us; skew 500 ppm
I20260810 15:21:25.495965 15354 webserver.cc:460] Webserver started at http://127.4.64.67:46123/ using document root <none> and password file <none>
I20260810 15:21:25.496271 15354 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.496357 15354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.496539 15354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.497967 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "f830bda144f148328e4ebdf029b8184e"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "fb6357ca04bc2c2208a9087f0fd11032"
server_key_iv: "b746675dacd58afd2c60836316523277"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.498378 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "f830bda144f148328e4ebdf029b8184e"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "fb6357ca04bc2c2208a9087f0fd11032"
server_key_iv: "b746675dacd58afd2c60836316523277"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.499890 15354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:25.500874 15373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.501065 15354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:25.501185 15354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "f830bda144f148328e4ebdf029b8184e"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "fb6357ca04bc2c2208a9087f0fd11032"
server_key_iv: "b746675dacd58afd2c60836316523277"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.501338 15354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.515798 15354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.516137 15354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.516296 15354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.516549 15354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:25.516911 15354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:25.516966 15354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.517009 15354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:25.517050 15354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.523367 15354 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:44869
I20260810 15:21:25.523538 15509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:44869 every 8 connection(s)
I20260810 15:21:25.523824 15354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:21:25.524459 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15354
I20260810 15:21:25.524557 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:21:25.524796 4353 external_mini_cluster.cc:1428] Setting key d1497de02e9606082283225525fb3a18
I20260810 15:21:25.529714 15032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.524763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:46845 (local address 127.4.64.126:45667)
0810 15:21:25.524996 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.524999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.525286 (+ 287us) server_negotiation.cc:407] Connection header received
0810 15:21:25.525959 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.525960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.525972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.526021 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.526707 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.527216 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.528245 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.528465 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.528554 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.529409 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:25.529410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:25.529411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:25.529416 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:25.529431 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:25.529435 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:25.529487 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:25.529583 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 15:21:25.529631 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260810 15:21:25.530004 15512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.524761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:45667 (local address 127.4.64.67:46845)
0810 15:21:25.525141 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:25.525184 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:25.525864 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:25.526091 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:25.526095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:25.526106 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:25.526615 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:25.526619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.527320 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.527321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:25.528150 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:25.528152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:25.528565 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:25.528566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:25.528707 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:25.529266 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:21:25.529266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:25.529295 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:21:25.529318 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:25.529493 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:25.529497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:25.529500 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:25.529583 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:21:25.529721 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":83,"threads_started":1}
I20260810 15:21:25.530586 15510 heartbeater.cc:347] Connected to a master server at 127.4.64.126:45667
I20260810 15:21:25.530714 15510 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:25.530910 15510 heartbeater.cc:511] Master 127.4.64.126:45667 requested a full tablet report, sending...
I20260810 15:21:25.531379 14984 ts_manager.cc:194] Registered new tserver with Master: f830bda144f148328e4ebdf029b8184e (127.4.64.67:44869)
I20260810 15:21:25.531756 14984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.67:46845
I20260810 15:21:25.537122 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 15:21:25.539342 15032 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:25.538740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.4.64.126:45667)
0810 15:21:25.538805 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.538807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.538871 (+ 64us) server_negotiation.cc:407] Connection header received
0810 15:21:25.538954 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.538954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.538967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.539022 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.539251 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44386: BlockingRecv error: recv got EOF from 127.0.0.1:44386 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:21:25.539356 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15055
I20260810 15:21:25.553066 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15205
I20260810 15:21:25.562430 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15354
I20260810 15:21:25.570734 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 14944
2026-08-10T15:21:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (682 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T15:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:25Z Disabled control of system clock
I20260810 15:21:25.603905 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:38367
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:21:25.689718 15526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.689960 15526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.689982 15526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.691087 15526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:25.691126 15526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.691139 15526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.691160 15526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:25.691172 15526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:25.692484 15526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:38367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.692694 15526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.692962 15526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:25.698269 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.698333 15532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.698360 15526 server_base.cc:1034] running on GCE node
W20260810 15:21:25.698273 15531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.698835 15526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.699115 15526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.700278 15526 hybrid_clock.cc:648] HybridClock initialized: now 1786375285700248 us; error 43 us; skew 500 ppm
I20260810 15:21:25.701596 15526 webserver.cc:460] Webserver started at http://127.4.64.126:34217/ using document root <none> and password file <none>
I20260810 15:21:25.701929 15526 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.701979 15526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.702148 15526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.703236 15526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "34c97ab82aa34f9b8d3417318d4e6654"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "84b7ca80affde816d2d8e9770de8a52f"
server_key_iv: "acbe31243916f612ee1111901f126e1a"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.703603 15526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "34c97ab82aa34f9b8d3417318d4e6654"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "84b7ca80affde816d2d8e9770de8a52f"
server_key_iv: "acbe31243916f612ee1111901f126e1a"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.705016 15526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:25.705871 15541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.706055 15526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:25.706146 15526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "34c97ab82aa34f9b8d3417318d4e6654"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "84b7ca80affde816d2d8e9770de8a52f"
server_key_iv: "acbe31243916f612ee1111901f126e1a"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.706202 15526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.710566 15526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.710845 15526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.710968 15526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.715339 15526 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:38367
I20260810 15:21:25.715418 15603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:38367 every 8 connection(s)
I20260810 15:21:25.715704 15526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:21:25.716420 15604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:25.719303 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15526
I20260810 15:21:25.719331 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654: Bootstrap starting.
I20260810 15:21:25.719442 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:21:25.719681 4353 external_mini_cluster.cc:1428] Setting key ae9de0aa85d7c23cf8f2c35d27c28f05
I20260810 15:21:25.720052 15604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:25.720422 15604 log.cc:824] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:25.721169 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654: No bootstrap required, opened a new log
I20260810 15:21:25.722570 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.720326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.4.64.126:38367)
0810 15:21:25.720569 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.720578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.720617 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:21:25.721072 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.721079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.721119 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.721315 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.722097 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:25.722101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:25.722102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:25.722106 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:25.722165 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:25.722176 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:25.722283 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:25.722325 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:21:25.722389 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260810 15:21:25.722951 15604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c97ab82aa34f9b8d3417318d4e6654" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38367 } }
I20260810 15:21:25.723107 15604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:25.723155 15604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c97ab82aa34f9b8d3417318d4e6654, State: Initialized, Role: FOLLOWER
I20260810 15:21:25.723306 15604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c97ab82aa34f9b8d3417318d4e6654" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38367 } }
I20260810 15:21:25.723412 15604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:21:25.723459 15604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:21:25.723507 15604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:25.724354 15604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c97ab82aa34f9b8d3417318d4e6654" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38367 } }
I20260810 15:21:25.724475 15604 leader_election.cc:302] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34c97ab82aa34f9b8d3417318d4e6654; no voters:
I20260810 15:21:25.724664 15604 leader_election.cc:288] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:21:25.724813 15610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:25.724980 15610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [term 1 LEADER]: Becoming Leader. State: Replica: 34c97ab82aa34f9b8d3417318d4e6654, State: Running, Role: LEADER
I20260810 15:21:25.724999 15604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:21:25.725085 15610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c97ab82aa34f9b8d3417318d4e6654" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38367 } }
I20260810 15:21:25.725594 15611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34c97ab82aa34f9b8d3417318d4e6654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c97ab82aa34f9b8d3417318d4e6654" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38367 } } }
I20260810 15:21:25.725598 15612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34c97ab82aa34f9b8d3417318d4e6654. Latest consensus state: current_term: 1 leader_uuid: "34c97ab82aa34f9b8d3417318d4e6654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34c97ab82aa34f9b8d3417318d4e6654" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38367 } } }
I20260810 15:21:25.725778 15611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:25.725805 15612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654 [sys.catalog]: This master's current role is: LEADER
W20260810 15:21:25.726949 15629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:21:25.727020 15629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:21:25.727083 15620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:25.727277 15620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:25.728909 15620 catalog_manager.cc:1347] Generated new cluster ID: f802c2bfce684e69ad55a16690f4503d
I20260810 15:21:25.728969 15620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:21:25.751531 15620 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:21:25.751775 15620 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:25.754927 15620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34c97ab82aa34f9b8d3417318d4e6654: Generated new TSK 0
I20260810 15:21:25.755136 15620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:21:25.760545 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38367
--builtin_ntp_servers=127.4.64.84:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:21:25.849314 15632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.849531 15632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.849555 15632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.850818 15632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.850867 15632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.850894 15632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:21:25.852387 15632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.852694 15632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.853042 15632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:25.859053 15642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.859097 15640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.859069 15638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.859503 15632 server_base.cc:1034] running on GCE node
I20260810 15:21:25.859773 15632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.860040 15632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.861225 15632 hybrid_clock.cc:648] HybridClock initialized: now 1786375285861193 us; error 40 us; skew 500 ppm
I20260810 15:21:25.862843 15632 webserver.cc:460] Webserver started at http://127.4.64.65:42965/ using document root <none> and password file <none>
I20260810 15:21:25.863171 15632 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.863252 15632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.863423 15632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.864559 15632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "8cff95a2ba9d49bdba01d48cf6b2f402"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "bcfa0110a3f3ab5a9aa32cda586a19ef"
server_key_iv: "b01b1ab34b9ec04d8039debe75516b9f"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.864936 15632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "8cff95a2ba9d49bdba01d48cf6b2f402"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "bcfa0110a3f3ab5a9aa32cda586a19ef"
server_key_iv: "b01b1ab34b9ec04d8039debe75516b9f"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.866496 15632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:21:25.867470 15648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.867722 15632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:25.867852 15632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "8cff95a2ba9d49bdba01d48cf6b2f402"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "bcfa0110a3f3ab5a9aa32cda586a19ef"
server_key_iv: "b01b1ab34b9ec04d8039debe75516b9f"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.867938 15632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.871506 15632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.871802 15632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.871953 15632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.872169 15632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:25.872502 15632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:25.872558 15632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.872599 15632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:25.872627 15632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.878579 15632 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:33159
I20260810 15:21:25.878661 15783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:33159 every 8 connection(s)
I20260810 15:21:25.879026 15632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:21:25.882664 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.879857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:60019 (local address 127.4.64.126:38367)
0810 15:21:25.879990 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:25.879993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:25.880306 (+ 313us) server_negotiation.cc:407] Connection header received
0810 15:21:25.880828 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:25.880829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:25.880836 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:25.880901 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:25.882334 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:25.882336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:25.882337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:25.882342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:25.882360 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:25.882365 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:25.882442 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:25.882555 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 15:21:25.882573 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 15:21:25.882896 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:25.879819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:38367 (local address 127.4.64.65:60019)
0810 15:21:25.880098 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:25.880135 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:25.880679 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:25.880981 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:25.880985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:25.880994 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:25.882109 (+ 1115us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:21:25.882109 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:25.882179 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:21:25.882213 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:25.882440 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:25.882447 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:25.882459 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 15:21:25.882568 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 15:21:25.882640 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":122,"threads_started":1}
I20260810 15:21:25.883486 15784 heartbeater.cc:347] Connected to a master server at 127.4.64.126:38367
I20260810 15:21:25.883596 15784 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:25.883805 15784 heartbeater.cc:511] Master 127.4.64.126:38367 requested a full tablet report, sending...
I20260810 15:21:25.884390 15560 ts_manager.cc:194] Registered new tserver with Master: 8cff95a2ba9d49bdba01d48cf6b2f402 (127.4.64.65:33159)
I20260810 15:21:25.886772 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15632
I20260810 15:21:25.886896 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:21:25.887188 4353 external_mini_cluster.cc:1428] Setting key 96d02b3a89d98170b08906f0724033c5
I20260810 15:21:25.888413 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38367
--builtin_ntp_servers=127.4.64.84:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:21:25.977216 15788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:25.977448 15788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:25.977473 15788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:25.978781 15788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:25.978837 15788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:25.978891 15788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:21:25.980309 15788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:25.980602 15788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:25.980939 15788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:25.986668 15795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.986686 15794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:25.986663 15797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:25.986948 15788 server_base.cc:1034] running on GCE node
I20260810 15:21:25.987238 15788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:25.987577 15788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:25.988785 15788 hybrid_clock.cc:648] HybridClock initialized: now 1786375285988757 us; error 33 us; skew 500 ppm
I20260810 15:21:25.990427 15788 webserver.cc:460] Webserver started at http://127.4.64.66:36679/ using document root <none> and password file <none>
I20260810 15:21:25.990715 15788 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:25.990797 15788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:25.990957 15788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:25.992110 15788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "d8fa38b6bc8946279a904fc0e23364e9"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "2eea15e74ca718cd796a93e897bfd3d2"
server_key_iv: "e062e68d0241ec916a9c65347bde78ca"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.992508 15788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "d8fa38b6bc8946279a904fc0e23364e9"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "2eea15e74ca718cd796a93e897bfd3d2"
server_key_iv: "e062e68d0241ec916a9c65347bde78ca"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.993954 15788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:21:25.994964 15803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.995242 15788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:25.995323 15788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "d8fa38b6bc8946279a904fc0e23364e9"
format_stamp: "Formatted at 2026-08-10 15:21:25 on dist-test-slave-9zdj"
server_key: "2eea15e74ca718cd796a93e897bfd3d2"
server_key_iv: "e062e68d0241ec916a9c65347bde78ca"
server_key_version: "encryptionkey@0"
I20260810 15:21:25.995386 15788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:25.998955 15788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:25.999204 15788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:25.999305 15788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:25.999501 15788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:25.999783 15788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:25.999815 15788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:25.999835 15788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:25.999847 15788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:26.005980 15788 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:42169
I20260810 15:21:26.006120 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:42169 every 8 connection(s)
I20260810 15:21:26.006500 15788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:21:26.009927 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:26.007378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:56221 (local address 127.4.64.126:38367)
0810 15:21:26.007445 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:26.007447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:26.007676 (+ 229us) server_negotiation.cc:407] Connection header received
0810 15:21:26.008241 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:26.008242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:26.008249 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:26.008309 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:26.009725 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:26.009727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:26.009727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:26.009732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:26.009746 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:26.009750 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:26.009802 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:26.009855 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 15:21:26.009868 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:21:26.010135 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:26.007323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:38367 (local address 127.4.64.66:56221)
0810 15:21:26.007536 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:26.007573 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:26.008057 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:26.008404 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:26.008408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:26.008418 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:26.009510 (+ 1092us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:21:26.009510 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:26.009540 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:21:26.009570 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:26.009800 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:26.009803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:26.009806 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:26.009866 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:21:26.009919 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260810 15:21:26.010860 15939 heartbeater.cc:347] Connected to a master server at 127.4.64.126:38367
I20260810 15:21:26.010953 15939 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:26.011075 15939 heartbeater.cc:511] Master 127.4.64.126:38367 requested a full tablet report, sending...
I20260810 15:21:26.011430 15560 ts_manager.cc:194] Registered new tserver with Master: d8fa38b6bc8946279a904fc0e23364e9 (127.4.64.66:42169)
I20260810 15:21:26.015255 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15788
I20260810 15:21:26.015407 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:21:26.015578 4353 external_mini_cluster.cc:1428] Setting key 04c03fcd668d32e75340b9c2bd95f9f8
I20260810 15:21:26.016644 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38367
--builtin_ntp_servers=127.4.64.84:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:21:26.103622 15944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:26.103879 15944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:26.103907 15944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:26.105181 15944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:26.105232 15944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:26.105260 15944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:21:26.106858 15944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:26.107112 15944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:26.107400 15944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:26.113416 15950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:26.113454 15949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:26.113461 15952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:26.113790 15944 server_base.cc:1034] running on GCE node
I20260810 15:21:26.114123 15944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:26.114463 15944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:26.115665 15944 hybrid_clock.cc:648] HybridClock initialized: now 1786375286115629 us; error 52 us; skew 500 ppm
I20260810 15:21:26.117442 15944 webserver.cc:460] Webserver started at http://127.4.64.67:41271/ using document root <none> and password file <none>
I20260810 15:21:26.117830 15944 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:26.117899 15944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:26.118052 15944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:26.119211 15944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "0cce61c8e563468fb861b3c59e8efed5"
format_stamp: "Formatted at 2026-08-10 15:21:26 on dist-test-slave-9zdj"
server_key: "aa71f9120344925cc38cd267269b519f"
server_key_iv: "d7429fed0429f97374a18ef7935560cb"
server_key_version: "encryptionkey@0"
I20260810 15:21:26.119612 15944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "0cce61c8e563468fb861b3c59e8efed5"
format_stamp: "Formatted at 2026-08-10 15:21:26 on dist-test-slave-9zdj"
server_key: "aa71f9120344925cc38cd267269b519f"
server_key_iv: "d7429fed0429f97374a18ef7935560cb"
server_key_version: "encryptionkey@0"
I20260810 15:21:26.121109 15944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:21:26.122125 15959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:26.122390 15944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:21:26.122509 15944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "0cce61c8e563468fb861b3c59e8efed5"
format_stamp: "Formatted at 2026-08-10 15:21:26 on dist-test-slave-9zdj"
server_key: "aa71f9120344925cc38cd267269b519f"
server_key_iv: "d7429fed0429f97374a18ef7935560cb"
server_key_version: "encryptionkey@0"
I20260810 15:21:26.122593 15944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:26.126446 15944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:26.126751 15944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:26.126899 15944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:26.127122 15944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:26.127468 15944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:26.127526 15944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:26.127568 15944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:26.127596 15944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:26.133867 15944 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:41193
I20260810 15:21:26.133915 16099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:41193 every 8 connection(s)
I20260810 15:21:26.134337 15944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:21:26.137462 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:26.135178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:58987 (local address 127.4.64.126:38367)
0810 15:21:26.135255 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:26.135258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:26.135500 (+ 242us) server_negotiation.cc:407] Connection header received
0810 15:21:26.135983 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:26.135985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:26.135991 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:26.136051 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:26.137231 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:26.137232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:26.137233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:21:26.137237 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:26.137253 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:21:26.137257 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:26.137308 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:26.137393 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:21:26.137406 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:21:26.137709 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:26.135170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:38367 (local address 127.4.64.67:58987)
0810 15:21:26.135364 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:26.135400 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:26.135882 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:26.136132 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:26.136136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:26.136145 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:26.137084 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:21:26.137085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:26.137113 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:21:26.137141 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:26.137320 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:26.137325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:26.137327 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:21:26.137391 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:21:26.137443 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260810 15:21:26.138226 16100 heartbeater.cc:347] Connected to a master server at 127.4.64.126:38367
I20260810 15:21:26.138309 16100 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:26.138482 16100 heartbeater.cc:511] Master 127.4.64.126:38367 requested a full tablet report, sending...
I20260810 15:21:26.138882 15560 ts_manager.cc:194] Registered new tserver with Master: 0cce61c8e563468fb861b3c59e8efed5 (127.4.64.67:41193)
I20260810 15:21:26.143200 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 15944
I20260810 15:21:26.143285 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:21:26.143496 4353 external_mini_cluster.cc:1428] Setting key 805bd338296eb876e9a6f84d0cb17bb5
I20260810 15:21:26.144673 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 15:21:26.146713 15608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:21:26.146230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.4.64.126:38367)
0810 15:21:26.146304 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:26.146306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:26.146317 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:26.146439 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:26.146441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:26.146446 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:26.146503 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:26.146643 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43068: BlockingRecv error: recv got EOF from 127.0.0.1:43068 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:21:26.146822 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15632
I20260810 15:21:26.159278 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15788
I20260810 15:21:26.168164 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15944
I20260810 15:21:26.176316 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 15526
2026-08-10T15:21:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (606 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:21:26 dist-test-slave-9zdj krb5kdc[16112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:21:26 dist-test-slave-9zdj krb5kdc[16112](info): set up 2 sockets
Aug 10 15:21:26 dist-test-slave-9zdj krb5kdc[16112](info): commencing operation
krb5kdc: starting...
W20260810 15:21:27.317955 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:21:27.469847 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:46345
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:27.567318 16127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:27.567617 16127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:27.567657 16127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:27.568903 16127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:21:27.568993 16127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:27.569017 16127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:27.569041 16127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:21:27.569072 16127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:21:27.570709 16127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:27.571043 16127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:27.571312 16127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:27.576906 16135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:27.576974 16127 server_base.cc:1034] running on GCE node
W20260810 15:21:27.576927 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:27.576931 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:27.577416 16127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:27.577847 16127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:27.579046 16127 hybrid_clock.cc:648] HybridClock initialized: now 1786375287579025 us; error 57 us; skew 500 ppm
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:27.582185 16127 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:21:27.582748 16127 webserver.cc:460] Webserver started at http://127.4.64.126:40797/ using document root <none> and password file <none>
I20260810 15:21:27.582983 16127 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:27.583030 16127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:27.583140 16127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:27.584321 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "dfff65befc0f4c6aa15e386b52f3d2ef"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "8c72491fabecabc41ff26bdb98c20d8b"
server_key_iv: "e991b9b679d548ce500d2f8bc765da6f"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.584745 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "dfff65befc0f4c6aa15e386b52f3d2ef"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "8c72491fabecabc41ff26bdb98c20d8b"
server_key_iv: "e991b9b679d548ce500d2f8bc765da6f"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.586359 16127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:21:27.587319 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.587524 16127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:27.587651 16127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "dfff65befc0f4c6aa15e386b52f3d2ef"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "8c72491fabecabc41ff26bdb98c20d8b"
server_key_iv: "e991b9b679d548ce500d2f8bc765da6f"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.587720 16127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:27.600692 16127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:27.603600 16127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:27.603750 16127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:27.608718 16127 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:46345
I20260810 15:21:27.608759 16204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:46345 every 8 connection(s)
I20260810 15:21:27.609197 16127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:21:27.610442 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:27.613094 16205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef: Bootstrap starting.
I20260810 15:21:27.613823 16205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:27.614128 16205 log.cc:824] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:27.614939 16205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef: No bootstrap required, opened a new log
I20260810 15:21:27.616859 16205 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46345 } }
I20260810 15:21:27.617025 16205 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:27.617053 16205 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff65befc0f4c6aa15e386b52f3d2ef, State: Initialized, Role: FOLLOWER
I20260810 15:21:27.617184 16205 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46345 } }
I20260810 15:21:27.617286 16205 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:21:27.617331 16205 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:21:27.617388 16205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:27.617738 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 16127
I20260810 15:21:27.617864 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:21:27.618053 4353 external_mini_cluster.cc:1428] Setting key a658633581c681ee35d841f1b2e827a1
I20260810 15:21:27.618809 16205 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46345 } }
I20260810 15:21:27.618947 16205 leader_election.cc:302] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfff65befc0f4c6aa15e386b52f3d2ef; no voters:
I20260810 15:21:27.619141 16205 leader_election.cc:288] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:21:27.619167 16211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:27.619360 16211 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [term 1 LEADER]: Becoming Leader. State: Replica: dfff65befc0f4c6aa15e386b52f3d2ef, State: Running, Role: LEADER
I20260810 15:21:27.619448 16205 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:21:27.619506 16211 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46345 } }
I20260810 15:21:27.619962 16213 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfff65befc0f4c6aa15e386b52f3d2ef. Latest consensus state: current_term: 1 leader_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46345 } } }
I20260810 15:21:27.620045 16213 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:27.620217 16212 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfff65befc0f4c6aa15e386b52f3d2ef" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 46345 } } }
I20260810 15:21:27.620272 16212 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef [sys.catalog]: This master's current role is: LEADER
I20260810 15:21:27.620812 16216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:21:27.621515 16216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:21:27.623960 16216 catalog_manager.cc:1347] Generated new cluster ID: 42b5962e870b47e98ad7c881773cf665
I20260810 15:21:27.624024 16216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:27.630827 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:27.618637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.4.64.126:46345)
0810 15:21:27.618882 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:27.618896 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:27.618955 (+ 59us) server_negotiation.cc:407] Connection header received
0810 15:21:27.619108 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:27.619117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:27.619189 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:27.619364 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:27.620611 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.622148 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.623057 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.623670 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.623829 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:27.626809 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:27.626814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:27.626814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:27.626828 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:27.628977 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:27.629397 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:27.629400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:27.629401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:27.629522 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:27.629761 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:27.629762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:27.629763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:27.630147 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:27.630343 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:27.630419 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:21:27.630580 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
I20260810 15:21:27.632505 16216 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:21:27.633219 16216 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:21:27.643812 16216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef: Generated new TSK 0
I20260810 15:21:27.644076 16216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:21:27.685421 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46345
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:27.774048 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:27.774264 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:27.774290 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:27.775485 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:27.775530 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:27.775558 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:21:27.777115 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:27.777438 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:27.777747 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:27.783540 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:27.783564 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:27.783516 16245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:27.783779 16233 server_base.cc:1034] running on GCE node
I20260810 15:21:27.784087 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:27.784449 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:27.785713 16233 hybrid_clock.cc:648] HybridClock initialized: now 1786375287785672 us; error 50 us; skew 500 ppm
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:27.788597 16233 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:21:27.789109 16233 webserver.cc:460] Webserver started at http://127.4.64.65:38909/ using document root <none> and password file <none>
I20260810 15:21:27.789359 16233 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:27.789431 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:27.789602 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:27.790838 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "a3d379e820a44f6dbc2d5124223cbcc0"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "eb3516cc7ee96d47e6589c0f1699dad5"
server_key_iv: "591d2739d60f8fbe619d67788dbc2304"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.791271 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "a3d379e820a44f6dbc2d5124223cbcc0"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "eb3516cc7ee96d47e6589c0f1699dad5"
server_key_iv: "591d2739d60f8fbe619d67788dbc2304"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.792865 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:27.794128 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.794431 16233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:21:27.794555 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "a3d379e820a44f6dbc2d5124223cbcc0"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "eb3516cc7ee96d47e6589c0f1699dad5"
server_key_iv: "591d2739d60f8fbe619d67788dbc2304"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.794636 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:27.804278 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:27.806864 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:27.807061 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:27.807365 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:27.807770 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:27.807834 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.807876 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:27.807904 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.814323 16233 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:41365
I20260810 15:21:27.814400 16381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:41365 every 8 connection(s)
I20260810 15:21:27.814770 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:21:27.821796 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 16233
I20260810 15:21:27.821893 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:27.822160 4353 external_mini_cluster.cc:1428] Setting key c11f3ce654c3476dcc72b6253cb3f0ff
I20260810 15:21:27.825654 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:27.815601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:52677 (local address 127.4.64.126:46345)
0810 15:21:27.815683 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:27.815685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:27.815927 (+ 242us) server_negotiation.cc:407] Connection header received
0810 15:21:27.816457 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:27.816458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:27.816481 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:27.816537 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:27.817520 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.818195 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.819491 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.819702 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.819765 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:27.822468 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:27.822469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:27.822470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:27.822482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:27.824105 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:27.824563 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:27.824565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:27.824565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:27.824633 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:27.824910 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:27.824911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:27.824911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:27.825039 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:27.825125 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:27.825437 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 15:21:27.825532 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 15:21:27.825872 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:27.815591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:46345 (local address 127.4.64.65:52677)
0810 15:21:27.815785 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:27.815825 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:27.816311 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:27.816595 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:27.816599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:27.817183 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:27.817402 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:27.817407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.818321 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.818322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:27.819403 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:27.819406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.819786 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.819787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:27.819904 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:27.820616 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:27.820625 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:27.822368 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:27.824202 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:27.824212 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:27.824215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:27.824480 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:27.824714 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:27.824714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:27.824715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:27.824841 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:27.825127 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:27.825136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:27.825290 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 15:21:27.825413 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 15:21:27.825591 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260810 15:21:27.826535 16382 heartbeater.cc:347] Connected to a master server at 127.4.64.126:46345
I20260810 15:21:27.826648 16382 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:27.826841 16382 heartbeater.cc:511] Master 127.4.64.126:46345 requested a full tablet report, sending...
I20260810 15:21:27.827417 16158 ts_manager.cc:194] Registered new tserver with Master: a3d379e820a44f6dbc2d5124223cbcc0 (127.4.64.65:41365)
I20260810 15:21:27.828066 16158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:52677
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:21:27.854672 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46345
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:27.946251 16387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:27.946486 16387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:27.946514 16387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:27.947775 16387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:27.947831 16387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:27.947860 16387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:21:27.949425 16387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:27.949735 16387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:27.950006 16387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:27.955628 16395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:27.955647 16399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:27.955649 16396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:27.956017 16387 server_base.cc:1034] running on GCE node
I20260810 15:21:27.956225 16387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:27.956513 16387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:27.957690 16387 hybrid_clock.cc:648] HybridClock initialized: now 1786375287957659 us; error 36 us; skew 500 ppm
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:27.960287 16387 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:21:27.960709 16387 webserver.cc:460] Webserver started at http://127.4.64.66:43857/ using document root <none> and password file <none>
I20260810 15:21:27.960922 16387 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:27.960965 16387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:27.961099 16387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:27.962229 16387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "8886881309b140dc885c882229138e09"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "9069071a143be62afcdf3ed448a58b56"
server_key_iv: "25c577cfe0767fbe5ef4179c7b7a45e2"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.962594 16387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "8886881309b140dc885c882229138e09"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "9069071a143be62afcdf3ed448a58b56"
server_key_iv: "25c577cfe0767fbe5ef4179c7b7a45e2"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.963986 16387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:21:27.964805 16406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.964990 16387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:21:27.965108 16387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "8886881309b140dc885c882229138e09"
format_stamp: "Formatted at 2026-08-10 15:21:27 on dist-test-slave-9zdj"
server_key: "9069071a143be62afcdf3ed448a58b56"
server_key_iv: "25c577cfe0767fbe5ef4179c7b7a45e2"
server_key_version: "encryptionkey@0"
I20260810 15:21:27.965198 16387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:27.980660 16387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:27.982913 16387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:27.983103 16387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:27.983402 16387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:27.983754 16387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:27.983814 16387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.983851 16387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:27.983871 16387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:27.990090 16387 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:37871
I20260810 15:21:27.990168 16540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:37871 every 8 connection(s)
I20260810 15:21:27.990733 16387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:21:27.991647 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 16387
I20260810 15:21:27.991806 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:21:27.992048 4353 external_mini_cluster.cc:1428] Setting key ba432d303e11cc00d6f514fe628fa17c
Aug 10 15:21:27 dist-test-slave-9zdj krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375287, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:28.002614 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:27.991645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:40123 (local address 127.4.64.126:46345)
0810 15:21:27.991753 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:27.991756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:27.992105 (+ 349us) server_negotiation.cc:407] Connection header received
0810 15:21:27.992841 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:27.992842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:27.992852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:27.992959 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:27.994137 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.994931 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.996073 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.996311 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.996380 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:27.999169 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:27.999171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:27.999172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:27.999180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:28.000979 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:28.001450 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:28.001451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:28.001452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:28.001518 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:28.001863 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:28.001865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:28.001865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:28.001998 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:28.002071 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.002406 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.002504 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:21:28.002816 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:27.991630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:46345 (local address 127.4.64.66:40123)
0810 15:21:27.991928 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:27.991978 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:27.992710 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:27.993076 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:27.993081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:27.993808 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:27.994033 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:27.994046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.995038 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.995038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:27.995979 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:27.995982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:27.996415 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:27.996416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:27.996580 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:27.997217 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:27.997229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:27.999061 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:28.001092 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:28.001101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:28.001104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:28.001363 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:28.001589 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:28.001590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:28.001591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:28.001784 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:28.002074 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:28.002082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:28.002261 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.002367 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.002548 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
I20260810 15:21:28.003373 16541 heartbeater.cc:347] Connected to a master server at 127.4.64.126:46345
I20260810 15:21:28.003476 16541 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:28.003688 16541 heartbeater.cc:511] Master 127.4.64.126:46345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
I20260810 15:21:28.004130 16158 ts_manager.cc:194] Registered new tserver with Master: 8886881309b140dc885c882229138e09 (127.4.64.66:37871)
I20260810 15:21:28.004542 16158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:40123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:21:28.026263 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46345
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:42263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:21:28.113979 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:21:28.114234 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:21:28.114288 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:21:28.115705 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:21:28.115787 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:21:28.115834 16546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:21:28.117489 16546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:42263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:46345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:21:28.117843 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:21:28.118112 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:21:28.124034 16552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:28.124011 16555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:21:28.124014 16553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:21:28.124519 16546 server_base.cc:1034] running on GCE node
I20260810 15:21:28.124707 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:21:28.124926 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:21:28.126094 16546 hybrid_clock.cc:648] HybridClock initialized: now 1786375288126080 us; error 32 us; skew 500 ppm
Aug 10 15:21:28 dist-test-slave-9zdj krb5kdc[16112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375288, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:21:28.128916 16546 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:21:28.129382 16546 webserver.cc:460] Webserver started at http://127.4.64.67:37373/ using document root <none> and password file <none>
I20260810 15:21:28.129695 16546 fs_manager.cc:359] Metadata directory not provided
I20260810 15:21:28.129770 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:21:28.129879 16546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:21:28.131359 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "eb720dad13724559945dec728d00e6e9"
format_stamp: "Formatted at 2026-08-10 15:21:28 on dist-test-slave-9zdj"
server_key: "ebde77695239e83535b1f19c3be442b8"
server_key_iv: "d0bf2b4738f30e4dc38a8a35df4f1f63"
server_key_version: "encryptionkey@0"
I20260810 15:21:28.131752 16546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "eb720dad13724559945dec728d00e6e9"
format_stamp: "Formatted at 2026-08-10 15:21:28 on dist-test-slave-9zdj"
server_key: "ebde77695239e83535b1f19c3be442b8"
server_key_iv: "d0bf2b4738f30e4dc38a8a35df4f1f63"
server_key_version: "encryptionkey@0"
I20260810 15:21:28.133292 16546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:21:28.134183 16562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:28.134367 16546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:21:28.134469 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "eb720dad13724559945dec728d00e6e9"
format_stamp: "Formatted at 2026-08-10 15:21:28 on dist-test-slave-9zdj"
server_key: "ebde77695239e83535b1f19c3be442b8"
server_key_iv: "d0bf2b4738f30e4dc38a8a35df4f1f63"
server_key_version: "encryptionkey@0"
I20260810 15:21:28.134529 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:21:28.144654 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:21:28.146823 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:21:28.146952 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:21:28.147147 16546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:21:28.147418 16546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:21:28.147451 16546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:28.147475 16546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:21:28.147487 16546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:21:28.153658 16546 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43823
I20260810 15:21:28.153705 16691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43823 every 8 connection(s)
I20260810 15:21:28.154062 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:21:28 dist-test-slave-9zdj krb5kdc[16112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375288, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:21:28.162962 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 16546
I20260810 15:21:28.163094 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:21:28.163307 4353 external_mini_cluster.cc:1428] Setting key c1f45d437813c21f1f9bdbb611ce6892
I20260810 15:21:28.164244 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.154907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:41697 (local address 127.4.64.126:46345)
0810 15:21:28.154973 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.154975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.155168 (+ 193us) server_negotiation.cc:407] Connection header received
0810 15:21:28.155740 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.155741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.155751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.155813 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:28.156751 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.157251 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.158343 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.158546 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.158603 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.160865 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:28.160866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:28.160866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:28.160873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:28.162406 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:28.162936 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:28.162937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:28.162938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:28.163003 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:28.163355 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:28.163356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:28.163356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:28.163532 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:28.163618 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.164027 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.164149 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:21:28.164569 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.154864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:46345 (local address 127.4.64.67:41697)
0810 15:21:28.155034 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.155078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:28.155650 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:28.155867 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:28.155871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:28.156471 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:21:28.156675 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.156679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.157338 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.157339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.158262 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.158265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.158646 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.158647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.158793 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.159328 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:21:28.159336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:21:28.160769 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:21:28.162501 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:28.162509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:28.162511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:28.162823 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:28.163115 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:21:28.163116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:21:28.163117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:21:28.163258 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:21:28.163635 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:21:28.163644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:21:28.163845 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.163965 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.164290 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260810 15:21:28.165040 16693 heartbeater.cc:347] Connected to a master server at 127.4.64.126:46345
I20260810 15:21:28.165187 16693 heartbeater.cc:464] Registering TS with master...
I20260810 15:21:28.165378 16693 heartbeater.cc:511] Master 127.4.64.126:46345 requested a full tablet report, sending...
I20260810 15:21:28.165824 16158 ts_manager.cc:194] Registered new tserver with Master: eb720dad13724559945dec728d00e6e9 (127.4.64.67:43823)
I20260810 15:21:28.166225 16158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:41697
I20260810 15:21:28.175599 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:21:28.183965 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.177019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.4.64.126:46345)
0810 15:21:28.177085 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.177088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.177202 (+ 114us) server_negotiation.cc:407] Connection header received
0810 15:21:28.177299 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.177300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.177312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.177371 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:21:28.178092 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.178612 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.179357 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.179661 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.179794 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.181030 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:21:28.181031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:21:28.181032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:21:28.181040 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:21:28.182591 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:28.183035 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:28.183036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:28.183037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:28.183104 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:21:28.183319 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:21:28.183320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:21:28.183320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:21:28.183518 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:21:28.183631 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.183756 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.183845 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:21:28.186376 16158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:21:28.188122 16158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:21:28.195768 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.190963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43823 (local address 127.0.0.1:33486)
0810 15:21:28.191658 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.191668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:28.191721 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:28.191968 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:28.191969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:28.191971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:28.192008 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.192009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.193256 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.193257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.194309 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.194312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.195346 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.195347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.195454 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.195475 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.195579 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.195655 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260810 15:21:28.195749 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.191202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.4.64.65:41365)
0810 15:21:28.191399 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.191404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.191416 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:21:28.191460 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.191462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.191478 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.191559 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:28.191775 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.192960 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.194575 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.195259 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.195326 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.195384 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.195606 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.195653 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260810 15:21:28.196156 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.191240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.4.64.67:43823)
0810 15:21:28.191441 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.191446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.191787 (+ 341us) server_negotiation.cc:407] Connection header received
0810 15:21:28.191845 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.191847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.191864 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.191941 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:28.192094 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.193143 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.194424 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.195190 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.195278 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.195351 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.196020 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.196068 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260810 15:21:28.195781 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.190925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41365 (local address 127.0.0.1:58336)
0810 15:21:28.191245 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.191258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:28.191319 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:28.191572 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:28.191575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:28.191578 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:28.191672 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.191675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.193075 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.193076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.194482 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.194486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.195363 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.195363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.195454 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.195475 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.195579 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.195702 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":122,"thread_start_us":82,"threads_started":1}
I20260810 15:21:28.196568 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.190917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:37871 (local address 127.0.0.1:55844)
0810 15:21:28.191136 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.191181 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:28.191255 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:28.191699 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:28.191700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:28.191703 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:28.191768 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.191770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.192920 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.192921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.194483 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.194486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.196310 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.196311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.196392 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.196398 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.196453 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.196500 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":62,"thread_start_us":75,"threads_started":1}
I20260810 15:21:28.196866 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.191051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.4.64.66:37871)
0810 15:21:28.191443 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.191447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.191458 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:21:28.191502 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.191504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.191520 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.191598 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:28.191834 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.192804 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.195425 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.196129 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.196205 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.196271 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.196748 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.196793 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":93,"threads_started":1}
I20260810 15:21:28.197122 16308 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6d64eb05dd422ea5cdc7ec059e1625 (DEFAULT_TABLE table=test-table [id=8ba5df64b38d46f29df3a817b0f2b325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:28.197124 16615 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6d64eb05dd422ea5cdc7ec059e1625 (DEFAULT_TABLE table=test-table [id=8ba5df64b38d46f29df3a817b0f2b325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:28.197664 16615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6d64eb05dd422ea5cdc7ec059e1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:28.197657 16308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6d64eb05dd422ea5cdc7ec059e1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:28.198364 16461 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6d64eb05dd422ea5cdc7ec059e1625 (DEFAULT_TABLE table=test-table [id=8ba5df64b38d46f29df3a817b0f2b325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:21:28.198731 16461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6d64eb05dd422ea5cdc7ec059e1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:21:28.201316 16713 tablet_bootstrap.cc:492] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Bootstrap starting.
I20260810 15:21:28.201921 16715 tablet_bootstrap.cc:492] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: Bootstrap starting.
I20260810 15:21:28.202221 16713 tablet_bootstrap.cc:654] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:28.202622 16715 tablet_bootstrap.cc:654] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:28.202860 16713 log.cc:824] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:28.203213 16715 log.cc:824] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:28.203578 16714 tablet_bootstrap.cc:492] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Bootstrap starting.
I20260810 15:21:28.203853 16713 tablet_bootstrap.cc:492] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: No bootstrap required, opened a new log
I20260810 15:21:28.203939 16713 ts_tablet_manager.cc:1397] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:21:28.204243 16714 tablet_bootstrap.cc:654] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Neither blocks nor log segments found. Creating new log.
I20260810 15:21:28.204777 16714 log.cc:824] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Log is configured to *not* fsync() on all Append() calls
I20260810 15:21:28.205940 16714 tablet_bootstrap.cc:492] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: No bootstrap required, opened a new log
I20260810 15:21:28.206039 16714 ts_tablet_manager.cc:1397] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:21:28.206128 16713 raft_consensus.cc:348] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.206276 16713 raft_consensus.cc:374] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:28.206298 16713 raft_consensus.cc:729] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8886881309b140dc885c882229138e09, State: Initialized, Role: FOLLOWER
I20260810 15:21:28.206445 16713 consensus_queue.cc:260] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.206795 16713 ts_tablet_manager.cc:1428] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 15:21:28.206837 16541 heartbeater.cc:503] Master 127.4.64.126:46345 was elected leader, sending a full tablet report...
I20260810 15:21:28.207417 16715 tablet_bootstrap.cc:492] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: No bootstrap required, opened a new log
I20260810 15:21:28.207511 16715 ts_tablet_manager.cc:1397] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 15:21:28.208187 16714 raft_consensus.cc:348] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.208366 16714 raft_consensus.cc:374] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:28.208393 16714 raft_consensus.cc:729] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb720dad13724559945dec728d00e6e9, State: Initialized, Role: FOLLOWER
I20260810 15:21:28.208528 16714 consensus_queue.cc:260] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.208841 16714 ts_tablet_manager.cc:1428] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:21:28.208912 16693 heartbeater.cc:503] Master 127.4.64.126:46345 was elected leader, sending a full tablet report...
I20260810 15:21:28.209774 16715 raft_consensus.cc:348] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.209978 16715 raft_consensus.cc:374] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:21:28.210026 16715 raft_consensus.cc:729] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d379e820a44f6dbc2d5124223cbcc0, State: Initialized, Role: FOLLOWER
I20260810 15:21:28.210151 16715 consensus_queue.cc:260] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.210453 16715 ts_tablet_manager.cc:1428] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:21:28.210620 16382 heartbeater.cc:503] Master 127.4.64.126:46345 was elected leader, sending a full tablet report...
W20260810 15:21:28.241482 16542 tablet.cc:2369] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:21:28.315627 16383 tablet.cc:2369] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:21:28.355569 16720 raft_consensus.cc:482] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:21:28.355729 16720 raft_consensus.cc:504] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.356266 16720 leader_election.cc:288] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8886881309b140dc885c882229138e09 (127.4.64.66:37871), a3d379e820a44f6dbc2d5124223cbcc0 (127.4.64.65:41365)
I20260810 15:21:28.359558 16710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.356618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:59087 (local address 127.4.64.65:41365)
0810 15:21:28.356711 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.356764 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.356857 (+ 93us) server_negotiation.cc:407] Connection header received
0810 15:21:28.356952 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.356953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.356966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.357041 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:28.357196 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.357746 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.358819 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.359191 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.359281 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.359292 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.359447 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.359485 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 15:21:28.359570 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.356520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41365 (local address 127.4.64.67:59087)
0810 15:21:28.356770 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.356780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:28.356874 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:28.357044 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:28.357045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:28.357047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:28.357124 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.357125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.357830 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.357831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.358724 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.358727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.359290 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.359291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.359370 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.359373 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.359442 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.359484 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":101,"thread_start_us":104,"threads_started":1}
I20260810 15:21:28.359838 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.356520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:37871 (local address 127.4.64.67:51917)
0810 15:21:28.356573 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.356589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 15:21:28.356663 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:21:28.356892 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:21:28.356893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:21:28.356895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:21:28.356992 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.356994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.357844 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.357845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.358751 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:21:28.358754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.359518 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.359519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:21:28.359603 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.359606 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:21:28.359686 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:21:28.359753 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":34}
I20260810 15:21:28.359931 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.356638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:51917 (local address 127.4.64.66:37871)
0810 15:21:28.356739 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.356741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.356751 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:21:28.356782 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.356783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.356805 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.356885 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:21:28.357089 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.357749 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.358950 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.359418 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.359516 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.359528 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.359822 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.359871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:21:28.359942 16330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d64eb05dd422ea5cdc7ec059e1625" candidate_uuid: "eb720dad13724559945dec728d00e6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" is_pre_election: true
I20260810 15:21:28.360101 16330 raft_consensus.cc:2454] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb720dad13724559945dec728d00e6e9 in term 0.
I20260810 15:21:28.360257 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d64eb05dd422ea5cdc7ec059e1625" candidate_uuid: "eb720dad13724559945dec728d00e6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8886881309b140dc885c882229138e09" is_pre_election: true
I20260810 15:21:28.360412 16568 leader_election.cc:302] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3d379e820a44f6dbc2d5124223cbcc0, eb720dad13724559945dec728d00e6e9; no voters:
I20260810 15:21:28.360450 16487 raft_consensus.cc:2454] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb720dad13724559945dec728d00e6e9 in term 0.
I20260810 15:21:28.360554 16720 raft_consensus.cc:2781] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:21:28.360652 16720 raft_consensus.cc:482] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:21:28.360685 16720 raft_consensus.cc:3037] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:28.361544 16720 raft_consensus.cc:504] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.361747 16720 leader_election.cc:288] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [CANDIDATE]: Term 1 election: Requested vote from peers 8886881309b140dc885c882229138e09 (127.4.64.66:37871), a3d379e820a44f6dbc2d5124223cbcc0 (127.4.64.65:41365)
I20260810 15:21:28.361959 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d64eb05dd422ea5cdc7ec059e1625" candidate_uuid: "eb720dad13724559945dec728d00e6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8886881309b140dc885c882229138e09"
I20260810 15:21:28.362040 16487 raft_consensus.cc:3037] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:28.362924 16487 raft_consensus.cc:2454] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb720dad13724559945dec728d00e6e9 in term 1.
I20260810 15:21:28.363140 16563 leader_election.cc:302] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8886881309b140dc885c882229138e09, eb720dad13724559945dec728d00e6e9; no voters:
I20260810 15:21:28.363099 16330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6d64eb05dd422ea5cdc7ec059e1625" candidate_uuid: "eb720dad13724559945dec728d00e6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3d379e820a44f6dbc2d5124223cbcc0"
I20260810 15:21:28.363250 16720 raft_consensus.cc:2781] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:21:28.363229 16330 raft_consensus.cc:3037] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:21:28.363441 16720 raft_consensus.cc:686] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 1 LEADER]: Becoming Leader. State: Replica: eb720dad13724559945dec728d00e6e9, State: Running, Role: LEADER
I20260810 15:21:28.363677 16720 consensus_queue.cc:237] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } }
I20260810 15:21:28.364262 16330 raft_consensus.cc:2454] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb720dad13724559945dec728d00e6e9 in term 1.
I20260810 15:21:28.364826 16158 catalog_manager.cc:5495] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb720dad13724559945dec728d00e6e9 (127.4.64.67). New cstate: current_term: 1 leader_uuid: "eb720dad13724559945dec728d00e6e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb720dad13724559945dec728d00e6e9" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 } health_report { overall_health: UNKNOWN } } }
I20260810 15:21:28.382476 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.379946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.4.64.67:43823)
0810 15:21:28.380015 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:21:28.380017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:21:28.380026 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:21:28.380065 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:21:28.380066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:21:28.380087 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:21:28.380166 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:21:28.380358 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.380845 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.381694 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:21:28.381910 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:21:28.381972 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:21:28.382114 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:21:28.382221 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:21:28.382280 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:21:28.382365 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:21:28.382390 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:21:28.384557 16487 raft_consensus.cc:1261] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 1 FOLLOWER]: Refusing update from remote peer eb720dad13724559945dec728d00e6e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:28.384599 16330 raft_consensus.cc:1261] T 1c6d64eb05dd422ea5cdc7ec059e1625 P a3d379e820a44f6dbc2d5124223cbcc0 [term 1 FOLLOWER]: Refusing update from remote peer eb720dad13724559945dec728d00e6e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:21:28.385017 16720 consensus_queue.cc:1035] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d379e820a44f6dbc2d5124223cbcc0" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:28.385197 16723 consensus_queue.cc:1035] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8886881309b140dc885c882229138e09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:21:28.387694 16727 mvcc.cc:205] Tried to move back new op lower bound from 7316993181219659776 to 7316993181138530304. Current Snapshot: MvccSnapshot[applied={T|T < 7316993181219659776}]
I20260810 15:21:28.388871 16728 mvcc.cc:205] Tried to move back new op lower bound from 7316993181219659776 to 7316993181138530304. Current Snapshot: MvccSnapshot[applied={T|T < 7316993181219659776}]
I20260810 15:21:28.388983 16729 mvcc.cc:205] Tried to move back new op lower bound from 7316993181219659776 to 7316993181138530304. Current Snapshot: MvccSnapshot[applied={T|T < 7316993181219659776 or (T in {7316993181219659776})}]
I20260810 15:21:28.390359 16158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:28.390461 16158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:21:28.391264 16158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfff65befc0f4c6aa15e386b52f3d2ef: Sending DeleteTablet for 3 replicas of tablet 1c6d64eb05dd422ea5cdc7ec059e1625
I20260810 15:21:28.391669 16461 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d64eb05dd422ea5cdc7ec059e1625 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:28 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:55844
I20260810 15:21:28.391712 16308 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d64eb05dd422ea5cdc7ec059e1625 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:28 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:58336
I20260810 15:21:28.391762 16615 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6d64eb05dd422ea5cdc7ec059e1625 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:21:28 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:33486
I20260810 15:21:28.392128 16739 tablet_replica.cc:330] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: stopping tablet replica
I20260810 15:21:28.392237 16739 raft_consensus.cc:2229] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:21:28.392237 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 16233
I20260810 15:21:28.392329 16739 raft_consensus.cc:2258] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:28.395141 16739 ts_tablet_manager.cc:1905] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:28.397287 16739 ts_tablet_manager.cc:1918] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:28.397394 16739 log.cc:1196] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/1c6d64eb05dd422ea5cdc7ec059e1625
I20260810 15:21:28.397758 16739 ts_tablet_manager.cc:1939] T 1c6d64eb05dd422ea5cdc7ec059e1625 P eb720dad13724559945dec728d00e6e9: Deleting consensus metadata
I20260810 15:21:28.398278 16146 catalog_manager.cc:4841] TS eb720dad13724559945dec728d00e6e9 (127.4.64.67:43823): tablet 1c6d64eb05dd422ea5cdc7ec059e1625 (table test-table [id=8ba5df64b38d46f29df3a817b0f2b325]) successfully deleted
I20260810 15:21:28.401762 16737 tablet_replica.cc:330] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: stopping tablet replica
I20260810 15:21:28.401911 16737 raft_consensus.cc:2229] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:21:28.402015 16737 raft_consensus.cc:2258] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:21:28.402674 16737 ts_tablet_manager.cc:1905] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:21:28.404950 16737 ts_tablet_manager.cc:1918] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:21:28.405048 16737 log.cc:1196] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/1c6d64eb05dd422ea5cdc7ec059e1625
I20260810 15:21:28.405370 16737 ts_tablet_manager.cc:1939] T 1c6d64eb05dd422ea5cdc7ec059e1625 P 8886881309b140dc885c882229138e09: Deleting consensus metadata
I20260810 15:21:28.405983 16144 catalog_manager.cc:4841] TS 8886881309b140dc885c882229138e09 (127.4.64.66:37871): tablet 1c6d64eb05dd422ea5cdc7ec059e1625 (table test-table [id=8ba5df64b38d46f29df3a817b0f2b325]) successfully deleted
W20260810 15:21:28.406925 16147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:41365 (error 108)
I20260810 15:21:28.407574 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:21:28.407413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41365 (local address 127.0.0.1:58344)
0810 15:21:28.407481 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 15:21:28.407528 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 15:21:28.407668 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 16387
W20260810 15:21:28.412671 16147 catalog_manager.cc:4584] TS a3d379e820a44f6dbc2d5124223cbcc0 (127.4.64.65:41365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c6d64eb05dd422ea5cdc7ec059e1625: Network error: Client connection negotiation failed: client connection to 127.4.64.65:41365: connect: Connection refused (error 111)
I20260810 15:21:28.417272 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 16546
I20260810 15:21:28.425923 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 16127
2026-08-10T15:21:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2247 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:21:28 dist-test-slave-9zdj krb5kdc[16743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:21:28 dist-test-slave-9zdj krb5kdc[16743](info): set up 2 sockets
Aug 10 15:21:28 dist-test-slave-9zdj krb5kdc[16743](info): commencing operation
krb5kdc: starting...
W20260810 15:21:29.578817 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:21:29 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:21:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/rangeradmin_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/rangeradmin_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/rangerlookup_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/rangerlookup_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
I20260810 15:21:29.773502 4353 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task97bH11/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/postgres -l logfile start
2026-08-10 15:21:32.833 UTC [16777] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 15:21:32.833 UTC [16777] LOG: listening on IPv4 address "127.4.64.84", port 36395
2026-08-10 15:21:32.838 UTC [16777] LOG: listening on Unix socket "/tmp/.s.PGSQL.36395"
2026-08-10 15:21:32.876 UTC [16780] LOG: database system was shut down at 2026-08-10 15:21:31 UTC
2026-08-10 15:21:32.882 UTC [16777] LOG: database system is ready to accept connections
I20260810 15:21:33.933657 4353 mini_postgres.cc:96] Postgres bound to 36395
2026-08-10 15:21:33.942 UTC [16788] FATAL: database "slave" does not exist
127.4.64.84:36395 - accepting connections
I20260810 15:21:33.943651 4353 mini_ranger.cc:161] Starting Ranger...
I20260810 15:21:33.961903 4353 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 15:21:34.579663 4353 mini_ranger.cc:87] Created ranger Postgres database
I20260810 15:21:34.579806 4353 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-admin
2026-08-10 15:21:35,044 [I] DB FLAVOR :POSTGRES
2026-08-10 15:21:35,045 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:21:35,045 [I] Checking connection..
2026-08-10 15:21:35,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 15:21:35,349 [I] Checking connection passed.
2026-08-10 15:21:35,350 [I] --------- Verifying version history table ---------
2026-08-10 15:21:35,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:21:35,599 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:21:35,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:21:35,873 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:21:35,873 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 15:21:35,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 15:21:36.144 UTC [16861] WARNING: there is no transaction in progress
2026-08-10 15:21:36,152 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 15:21:36,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:21:36,420 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 15:21:36,421 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:21:36,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 15:21:36,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 15:21:36,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9zdj.c.gcp-upstream.internal','N') ;"
2026-08-10 15:21:37,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:21:37,492 [I] Table x_portal_user does not exist in database ranger
2026-08-10 15:21:37,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:21:37,792 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 15:21:37,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:21:38,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:21:38,295 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 15:21:38,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 15:21:39.394 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.414 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.435 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.453 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.473 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.496 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.559 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.566 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.581 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.591 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.602 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.618 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.629 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.636 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.644 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:39.652 UTC [16983] WARNING: there is no transaction in progress
2026-08-10 15:21:40,459 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 15:21:40,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 15:21:40,720 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 15:21:40,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:21:41,002 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 15:21:41,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:21:41,284 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 15:21:41,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:21:41,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:21:41,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9zdj.c.gcp-upstream.internal';"
2026-08-10 15:21:42,124 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 15:21:42,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 15:21:42,403 [I] DB_PATCHES have already been applied
I20260810 15:21:42.409904 4353 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-admin:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task97bH11/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 15:21:42.409982 4353 mini_ranger.cc:193] Using host: 127.4.64.84
I20260810 15:21:42.414897 4353 mini_ranger.cc:227] Ranger admin URL: http://127.4.64.84:57697
Aug 10, 2026 3:21:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:21:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:21:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:21:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:21:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:21:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:21:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 15:21:43.716204 4353 mini_ranger.cc:160] Time spent starting Ranger: real 9.773s user 0.002s sys 0.011s
Aug 10, 2026 3:22:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:22:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 10, 2026 3:22:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:22:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:22:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 15:22:08.452922 4353 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/rangerkms_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/rangerkms_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.64.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.64.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 15:22:08 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375328, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 15:22:08.568800 4353 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 15:22:08.581436 4353 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 15:22:09.624169 4353 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 15:22:09.624254 4353 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms
I20260810 15:22:09.624272 4353 mini_ranger_kms.cc:227] Using postgres at 127.4.64.84:36395
2026-08-10 15:22:09,916 [I] DB FLAVOR :POSTGRES
2026-08-10 15:22:09,916 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:22:09,916 [I] Checking connection
2026-08-10 15:22:09,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 15:22:10,189 [I] connection success
2026-08-10 15:22:10,189 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 15:22:10,189 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 15:22:10,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 15:22:10,454 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 15:22:10,454 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:22:10,454 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 15:22:10,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:36395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 15:22:10,764 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 15:22:11.323705 4353 mini_ranger_kms.cc:310] Created kms service
I20260810 15:22:11.524029 4353 mini_ranger_kms.cc:326] Created kudu user
I20260810 15:22:11.569566 4353 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 15:22:11 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375328, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:22:11.981208 4353 mini_ranger_kms.cc:384] Added ranger policy
I20260810 15:22:11.981345 4353 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms:/tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 15:22:11.981367 4353 mini_ranger_kms.cc:240] Using host: 127.4.64.84
I20260810 15:22:11.986403 4353 mini_ranger_kms.cc:276] Ranger KMS PID: 17209
I20260810 15:22:11.986546 4353 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.64.84:58433
Aug 10, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 15:22:14 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375334, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.64.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:22:14.651619 4353 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.083s user 0.001s sys 0.246s
I20260810 15:22:14.651727 4353 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 15:22:14.651810 4353 mini_ranger_kms.cc:401] 127.4.64.84:58433/kms/v1/keys
Aug 10 15:22:15 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375334, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.64.84@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
Aug 10 15:22:15 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375334, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.64.84@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
Aug 10 15:22:17 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:22:17.786079 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35673
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:35673
--ranger_config_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:58433/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:22:17.905426 17266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:22:17.905753 17266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:22:17.905829 17266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:22:17.907150 17266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:22:17.907200 17266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:22:17.907217 17266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:22:17.907258 17266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:22:17.907287 17266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:22:17.909099 17266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.64.84:58433/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:35673
--ranger_config_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:22:17.909379 17266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:22:17.909710 17266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:22:17.915970 17272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:22:17.916096 17266 server_base.cc:1034] running on GCE node
W20260810 15:22:17.916185 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:22:17.916237 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:22:17.916672 17266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:22:17.917024 17266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:22:17.918195 17266 hybrid_clock.cc:648] HybridClock initialized: now 1786375337918172 us; error 44 us; skew 500 ppm
Aug 10 15:22:17 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375337, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:22:17.921782 17266 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:22:17.922320 17266 webserver.cc:460] Webserver started at http://127.4.64.126:40405/ using document root <none> and password file <none>
I20260810 15:22:17.922557 17266 fs_manager.cc:359] Metadata directory not provided
I20260810 15:22:17.922619 17266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:22:17.922794 17266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:22:17 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375337, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:22:17.975759 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "86aca668106c4f03821e26522358e35f"
format_stamp: "Formatted at 2026-08-10 15:22:17 on dist-test-slave-9zdj"
server_key: "d736e69854ceb5c84f03b4f0729b83e5"
server_key_iv: "d0e5d777030abf1e5d12cd014aac97a9"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:17.976209 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "86aca668106c4f03821e26522358e35f"
format_stamp: "Formatted at 2026-08-10 15:22:17 on dist-test-slave-9zdj"
server_key: "d736e69854ceb5c84f03b4f0729b83e5"
server_key_iv: "d0e5d777030abf1e5d12cd014aac97a9"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:17.978387 17266 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:22:17.993909 17283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:17.994107 17266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:22:17.994230 17266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "86aca668106c4f03821e26522358e35f"
format_stamp: "Formatted at 2026-08-10 15:22:17 on dist-test-slave-9zdj"
server_key: "d736e69854ceb5c84f03b4f0729b83e5"
server_key_iv: "d0e5d777030abf1e5d12cd014aac97a9"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:17.994361 17266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:22:18.010994 17266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:22:18.013403 17266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:22:18.013602 17266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:22:18.018853 17266 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:35673
I20260810 15:22:18.018887 17343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:35673 every 8 connection(s)
I20260810 15:22:18.019291 17266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:22:18.020150 17344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:22:18.022661 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f: Bootstrap starting.
I20260810 15:22:18.023305 17344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f: Neither blocks nor log segments found. Creating new log.
I20260810 15:22:18.023612 17344 log.cc:824] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:22:18.024430 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f: No bootstrap required, opened a new log
I20260810 15:22:18.026717 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 17266
I20260810 15:22:18.026628 17344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aca668106c4f03821e26522358e35f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35673 } }
I20260810 15:22:18.026819 17344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:22:18.026851 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:22:18.026891 17344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86aca668106c4f03821e26522358e35f, State: Initialized, Role: FOLLOWER
I20260810 15:22:18.027062 4353 external_mini_cluster.cc:1428] Setting key fd1cccb27ee49fe265299eda58b1a9cf
I20260810 15:22:18.027062 17344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aca668106c4f03821e26522358e35f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35673 } }
I20260810 15:22:18.027148 17344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:22:18.027171 17344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:22:18.027204 17344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:22:18.028014 17344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aca668106c4f03821e26522358e35f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35673 } }
I20260810 15:22:18.028149 17344 leader_election.cc:302] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86aca668106c4f03821e26522358e35f; no voters:
I20260810 15:22:18.028374 17344 leader_election.cc:288] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:22:18.028426 17349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:22:18.028702 17349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [term 1 LEADER]: Becoming Leader. State: Replica: 86aca668106c4f03821e26522358e35f, State: Running, Role: LEADER
I20260810 15:22:18.028767 17344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:22:18.029248 17349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aca668106c4f03821e26522358e35f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35673 } }
I20260810 15:22:18.030592 17351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86aca668106c4f03821e26522358e35f. Latest consensus state: current_term: 1 leader_uuid: "86aca668106c4f03821e26522358e35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aca668106c4f03821e26522358e35f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35673 } } }
I20260810 15:22:18.030754 17344 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:22:18.030833 17350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86aca668106c4f03821e26522358e35f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86aca668106c4f03821e26522358e35f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35673 } } }
I20260810 15:22:18.030903 17350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [sys.catalog]: This master's current role is: LEADER
I20260810 15:22:18.031222 17367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:22:18.030742 17351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f [sys.catalog]: This master's current role is: LEADER
I20260810 15:22:18.034581 17367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 15:22:18 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:22:18.037863 17367 catalog_manager.cc:1347] Generated new cluster ID: d4e89a9049954f5a8a4c5b67d7598007
I20260810 15:22:18.037956 17367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:22:18.042656 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:18.027625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.4.64.126:35673)
0810 15:22:18.027836 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:18.027847 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:18.027881 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:22:18.028004 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:18.028011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:18.028069 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:18.028224 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:22:18.029068 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:18.030723 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:18.033131 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:18.034686 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:18.034820 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:18.038315 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:22:18.038319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:22:18.038319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:22:18.038331 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:22:18.040481 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:18.040930 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:18.040934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:18.040934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:18.041095 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:18.041329 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:18.041330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:18.041330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:18.041706 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:22:18.041859 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:18.042162 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 15:22:18.042336 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260810 15:22:18.074934 17367 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:22:18.075759 17367 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:22:18.082645 17367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f: Generated new TSK 0
I20260810 15:22:18.082947 17367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:22:18.844756 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:58433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35673
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:22:18.940321 17395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:22:18.940589 17395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:22:18.940637 17395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:22:18.941942 17395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:22:18.941987 17395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:22:18.942014 17395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:22:18.943655 17395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.64.84:58433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:22:18.943949 17395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:22:18.944255 17395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:22:18.950184 17402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:22:18.950227 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:22:18.950207 17395 server_base.cc:1034] running on GCE node
W20260810 15:22:18.950237 17405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:22:18.950817 17395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:22:18.951125 17395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:22:18.952354 17395 hybrid_clock.cc:648] HybridClock initialized: now 1786375338952328 us; error 40 us; skew 500 ppm
Aug 10 15:22:18 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375338, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:22:18.955559 17395 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:22:18.956157 17395 webserver.cc:460] Webserver started at http://127.4.64.65:34869/ using document root <none> and password file <none>
I20260810 15:22:18.956416 17395 fs_manager.cc:359] Metadata directory not provided
I20260810 15:22:18.956471 17395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:22:18.956609 17395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:22:18 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375338, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:22:18.972599 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd"
format_stamp: "Formatted at 2026-08-10 15:22:18 on dist-test-slave-9zdj"
server_key: "ee4508c0def03e2bb96b3c0d41ecd74f"
server_key_iv: "701fd2175fa9f03bb761fd9b9ce71039"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:18.973208 17395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd"
format_stamp: "Formatted at 2026-08-10 15:22:18 on dist-test-slave-9zdj"
server_key: "ee4508c0def03e2bb96b3c0d41ecd74f"
server_key_iv: "701fd2175fa9f03bb761fd9b9ce71039"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:18.975248 17395 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 15:22:18.992960 17415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:18.995302 17395 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260810 15:22:18.995446 17395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd"
format_stamp: "Formatted at 2026-08-10 15:22:18 on dist-test-slave-9zdj"
server_key: "ee4508c0def03e2bb96b3c0d41ecd74f"
server_key_iv: "701fd2175fa9f03bb761fd9b9ce71039"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:18.995571 17395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:22:19.020255 17395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:22:19.023412 17395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:22:19.023619 17395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:22:19.023931 17395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:22:19.024416 17395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:22:19.024477 17395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.024513 17395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:22:19.024578 17395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.033026 17395 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:41243
I20260810 15:22:19.033668 17395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:22:19.042183 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 17395
I20260810 15:22:19.042330 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:22:19.042557 4353 external_mini_cluster.cc:1428] Setting key c46f22eaf4da1401934116276bc6fd65
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375338, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:22:19.049253 17545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:41243 every 8 connection(s)
I20260810 15:22:19.055047 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.038419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35673 (local address 127.4.64.65:40009)
0810 15:22:19.039015 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:19.039069 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:19.039801 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:19.041385 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:19.041390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:19.041731 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:22:19.043028 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.043037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.044249 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.044251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.045825 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.045828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.046743 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.046745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.046916 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.047561 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:22:19.047572 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:22:19.049784 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:22:19.052817 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:22:19.052825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:22:19.052828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:22:19.053200 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:22:19.053592 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:22:19.053593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:22:19.053593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:22:19.053793 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:22:19.054183 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:22:19.054191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:22:19.054428 (+ 237us) client_negotiation.cc:769] Sending connection context
0810 15:22:19.054549 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 15:22:19.054750 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":107,"threads_started":1}
I20260810 15:22:19.055606 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.038793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:40009 (local address 127.4.64.126:35673)
0810 15:22:19.041055 (+ 2262us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.041058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.041070 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:22:19.041107 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.041108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.041123 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.041203 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:22:19.043172 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.044096 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.045984 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.046584 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.046663 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.049955 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:22:19.049957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:22:19.049957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:22:19.049966 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:22:19.052695 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.053424 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.053425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.053426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.053505 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.053887 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.053888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.053889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.054075 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:22:19.054151 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.055365 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.055490 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2228,"thread_start_us":97,"threads_started":1}
I20260810 15:22:19.056701 17546 heartbeater.cc:347] Connected to a master server at 127.4.64.126:35673
I20260810 15:22:19.056881 17546 heartbeater.cc:464] Registering TS with master...
I20260810 15:22:19.057283 17546 heartbeater.cc:511] Master 127.4.64.126:35673 requested a full tablet report, sending...
I20260810 15:22:19.057898 17303 ts_manager.cc:194] Registered new tserver with Master: 1e6cfe57e5ad48c5bb2ac71e44b77bfd (127.4.64.65:41243)
I20260810 15:22:19.058719 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:40009
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
W20260810 15:22:19.145524 4353 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.64.66: real 0.102s user 0.001s sys 0.003s
I20260810 15:22:19.146055 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:58433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35673
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 15:22:19.290699 17558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:22:19.291121 17558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:22:19.291222 17558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:22:19.293296 17558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:22:19.293440 17558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:22:19.293552 17558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:22:19.296387 17558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.64.84:58433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:22:19.297013 17558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:22:19.297389 17558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:22:19.304860 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:22:19.305291 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:22:19.305462 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:22:19.306353 17558 server_base.cc:1034] running on GCE node
I20260810 15:22:19.306703 17558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:22:19.307008 17558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:22:19.308192 17558 hybrid_clock.cc:648] HybridClock initialized: now 1786375339308154 us; error 51 us; skew 500 ppm
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:22:19.312089 17558 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:22:19.312697 17558 webserver.cc:460] Webserver started at http://127.4.64.66:38485/ using document root <none> and password file <none>
I20260810 15:22:19.312969 17558 fs_manager.cc:359] Metadata directory not provided
I20260810 15:22:19.313021 17558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:22:19.313156 17558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:22:19.334882 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "4db1660f7a134da0a6306ab0c9f88451"
format_stamp: "Formatted at 2026-08-10 15:22:19 on dist-test-slave-9zdj"
server_key: "6b3ef1186da9100c4e8c95ab160678f8"
server_key_iv: "417bbaf8a62a5f0c2cc5076852c2fa0b"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:19.336472 17558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "4db1660f7a134da0a6306ab0c9f88451"
format_stamp: "Formatted at 2026-08-10 15:22:19 on dist-test-slave-9zdj"
server_key: "6b3ef1186da9100c4e8c95ab160678f8"
server_key_iv: "417bbaf8a62a5f0c2cc5076852c2fa0b"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:19.341229 17558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260810 15:22:19.355923 17576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.356163 17558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:22:19.356279 17558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "4db1660f7a134da0a6306ab0c9f88451"
format_stamp: "Formatted at 2026-08-10 15:22:19 on dist-test-slave-9zdj"
server_key: "6b3ef1186da9100c4e8c95ab160678f8"
server_key_iv: "417bbaf8a62a5f0c2cc5076852c2fa0b"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:19.356375 17558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:22:19.377250 17558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:22:19.380110 17558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:22:19.380364 17558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:22:19.380709 17558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:22:19.381253 17558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:22:19.381356 17558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.381433 17558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:22:19.381493 17558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.398612 17558 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:43731
I20260810 15:22:19.399338 17558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:22:19.401443 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 17558
I20260810 15:22:19.401556 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:22:19.401875 4353 external_mini_cluster.cc:1428] Setting key 4114db3247833a2664a6bf813c2c52d2
I20260810 15:22:19.405144 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:43731 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:22:19.435391 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.405725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:43105 (local address 127.4.64.126:35673)
0810 15:22:19.405775 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.405777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.413983 (+ 8206us) server_negotiation.cc:407] Connection header received
0810 15:22:19.414836 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.414838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.414854 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.414887 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:22:19.416017 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.416870 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.425865 (+ 8995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.426219 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.426285 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.430594 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:22:19.430596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:22:19.430597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:22:19.430607 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:22:19.432451 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.433811 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.433812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.433813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.433883 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.434351 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.434352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.434353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.434527 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:22:19.434585 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.435116 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.435247 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:22:19.436650 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.405398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35673 (local address 127.4.64.66:43105)
0810 15:22:19.413766 (+ 8368us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:19.413850 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:19.414673 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:19.415110 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:19.415115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:19.415411 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:22:19.415746 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.415753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.417115 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.417116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.421176 (+ 4060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.421180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.426423 (+ 5243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.426425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.426644 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.427904 (+ 1260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:22:19.427917 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:22:19.430431 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:22:19.432570 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:22:19.432581 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:22:19.432584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:22:19.432927 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:22:19.433971 (+ 1044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:22:19.433972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:22:19.433973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:22:19.434157 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:22:19.434626 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:22:19.434636 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:22:19.434931 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 15:22:19.436155 (+ 1224us) client_negotiation.cc:240] Negotiation successful
0810 15:22:19.436381 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8291,"thread_start_us":123,"threads_started":1}
I20260810 15:22:19.437640 17709 heartbeater.cc:347] Connected to a master server at 127.4.64.126:35673
I20260810 15:22:19.437806 17709 heartbeater.cc:464] Registering TS with master...
I20260810 15:22:19.438038 17709 heartbeater.cc:511] Master 127.4.64.126:35673 requested a full tablet report, sending...
I20260810 15:22:19.438576 17303 ts_manager.cc:194] Registered new tserver with Master: 4db1660f7a134da0a6306ab0c9f88451 (127.4.64.66:43731)
I20260810 15:22:19.439090 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:43105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:22:19.445068 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:58433/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35673
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:22:19.575330 17715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:22:19.575641 17715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:22:19.575685 17715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:22:19.577438 17715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:22:19.577522 17715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:22:19.577567 17715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:22:19.579883 17715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.64.84:58433/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:22:19.580221 17715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:22:19.580518 17715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:22:19.589154 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:22:19.589406 17726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:22:19.589646 17723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:22:19.591008 17715 server_base.cc:1034] running on GCE node
I20260810 15:22:19.591320 17715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:22:19.591624 17715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:22:19.592804 17715 hybrid_clock.cc:648] HybridClock initialized: now 1786375339592781 us; error 35 us; skew 500 ppm
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:22:19.596287 17715 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:22:19.596921 17715 webserver.cc:460] Webserver started at http://127.4.64.67:34359/ using document root <none> and password file <none>
I20260810 15:22:19.597216 17715 fs_manager.cc:359] Metadata directory not provided
I20260810 15:22:19.597278 17715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:22:19.597420 17715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:22:19.613508 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "3870f1189fa64ff681c1bad727b027ee"
format_stamp: "Formatted at 2026-08-10 15:22:19 on dist-test-slave-9zdj"
server_key: "440942fb4b08f3272d4a3668107a7614"
server_key_iv: "ab2e3634d251ad25f11c67b8936d72ab"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:19.614002 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "3870f1189fa64ff681c1bad727b027ee"
format_stamp: "Formatted at 2026-08-10 15:22:19 on dist-test-slave-9zdj"
server_key: "440942fb4b08f3272d4a3668107a7614"
server_key_iv: "ab2e3634d251ad25f11c67b8936d72ab"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:19.617833 17715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 15:22:19.632992 17735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.633199 17715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:22:19.633313 17715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "3870f1189fa64ff681c1bad727b027ee"
format_stamp: "Formatted at 2026-08-10 15:22:19 on dist-test-slave-9zdj"
server_key: "440942fb4b08f3272d4a3668107a7614"
server_key_iv: "ab2e3634d251ad25f11c67b8936d72ab"
server_key_version: "kuduclusterkey@0"
I20260810 15:22:19.633441 17715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:22:19.674404 17715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:22:19.676996 17715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:22:19.677161 17715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:22:19.677435 17715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:22:19.677827 17715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:22:19.677876 17715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.677914 17715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:22:19.677942 17715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:22:19.684792 17715 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:40403
I20260810 15:22:19.685278 17715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:22:19.685431 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 17715
I20260810 15:22:19.685534 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:22:19.685791 4353 external_mini_cluster.cc:1428] Setting key 6e2368d16122d90d07601c423a505c3e
I20260810 15:22:19.688228 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:40403 every 8 connection(s)
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:22:19.700173 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.689107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:42469 (local address 127.4.64.126:35673)
0810 15:22:19.689151 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.689153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.689481 (+ 328us) server_negotiation.cc:407] Connection header received
0810 15:22:19.690320 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.690321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.690334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.690365 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:22:19.691222 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.692039 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.693570 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.693911 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.693955 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.696645 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:22:19.696646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:22:19.696647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:22:19.696655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:22:19.698502 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.699035 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.699036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.699037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.699096 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.699385 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.699386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.699386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.699530 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:22:19.699570 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.699986 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.700072 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:22:19.700785 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.688811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35673 (local address 127.4.64.67:42469)
0810 15:22:19.689352 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:19.689398 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:19.690195 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:19.690556 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:19.690560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:19.690833 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:22:19.691113 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.691119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.692151 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.692152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.693465 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.693467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.694049 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.694050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.694206 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.694864 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:22:19.694875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:22:19.696526 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:22:19.698602 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:22:19.698610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:22:19.698612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:22:19.698946 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:22:19.699164 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:22:19.699165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:22:19.699165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:22:19.699309 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:22:19.699615 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:22:19.699621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:22:19.699829 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 15:22:19.700335 (+ 506us) client_negotiation.cc:240] Negotiation successful
0810 15:22:19.700520 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":94,"threads_started":1}
I20260810 15:22:19.701237 17868 heartbeater.cc:347] Connected to a master server at 127.4.64.126:35673
I20260810 15:22:19.701345 17868 heartbeater.cc:464] Registering TS with master...
I20260810 15:22:19.701550 17868 heartbeater.cc:511] Master 127.4.64.126:35673 requested a full tablet report, sending...
I20260810 15:22:19.701933 17303 ts_manager.cc:194] Registered new tserver with Master: 3870f1189fa64ff681c1bad727b027ee (127.4.64.67:40403)
I20260810 15:22:19.702440 17303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:42469
I20260810 15:22:19.708824 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:22:19.718802 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.710643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.4.64.126:35673)
0810 15:22:19.710691 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.710693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.710931 (+ 238us) server_negotiation.cc:407] Connection header received
0810 15:22:19.710985 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.710986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.710998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.711058 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:22:19.711796 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.712561 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.713707 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.714001 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.714051 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.715695 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:22:19.715696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:22:19.715697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:22:19.715711 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:22:19.717444 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.717912 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.717913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.717914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.717988 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:22:19.718214 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:22:19.718215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:22:19.718216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:22:19.718362 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:22:19.718459 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.718609 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.718696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:22:19.721124 17303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:22:19.723484 17303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:22:19.733726 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.728137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:40403 (local address 127.0.0.1:45930)
0810 15:22:19.728685 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:19.728729 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:19.728797 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:19.729540 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:19.729545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:19.729547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:19.729655 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.729658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.730989 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.730990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.732260 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.732263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.733349 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.733350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.733428 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.733441 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 15:22:19.733549 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:22:19.733606 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260810 15:22:19.733945 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.728591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.4.64.67:40403)
0810 15:22:19.729341 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.729346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.729358 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:22:19.729403 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.729405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.729424 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.729520 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:19.729753 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.730894 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.732394 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.733155 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.733220 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.733297 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.733830 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.733868 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":92,"threads_started":1}
I20260810 15:22:19.735067 17791 tablet_service.cc:1467] Processing CreateTablet for tablet cd80a8fdadb24880a87eefe9d4d8ca9c (DEFAULT_TABLE table=test-table [id=aff6c7a4985b4015b4b07f6ba4112129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:22:19.735531 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd80a8fdadb24880a87eefe9d4d8ca9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:22:19.741025 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.728147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41243 (local address 127.0.0.1:47364)
0810 15:22:19.729653 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:19.729665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:19.729725 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:19.729992 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:19.729993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:19.729995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:19.730042 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.730043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.731806 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.731807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.733088 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.733091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.740736 (+ 7645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.740737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.740822 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.740827 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:22:19.740882 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:22:19.740931 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260810 15:22:19.741600 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.728798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.4.64.65:41243)
0810 15:22:19.729209 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.729215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.729791 (+ 576us) server_negotiation.cc:407] Connection header received
0810 15:22:19.729832 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.729835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.729853 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.729926 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:19.730132 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.731705 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.733207 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.740541 (+ 7334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.740625 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.740692 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.741463 (+ 771us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.741520 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":78,"threads_started":1}
I20260810 15:22:19.742357 17476 tablet_service.cc:1467] Processing CreateTablet for tablet cd80a8fdadb24880a87eefe9d4d8ca9c (DEFAULT_TABLE table=test-table [id=aff6c7a4985b4015b4b07f6ba4112129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:22:19.742761 17476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd80a8fdadb24880a87eefe9d4d8ca9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:22:19.747182 17890 tablet_bootstrap.cc:492] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: Bootstrap starting.
I20260810 15:22:19.747885 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.728365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:43731 (local address 127.0.0.1:37484)
0810 15:22:19.734050 (+ 5685us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:19.734061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:19.734104 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:19.735092 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:19.735092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:19.735094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:19.735122 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.735123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.742595 (+ 7472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.742596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.743761 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:19.743765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.747600 (+ 3835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.747602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:19.747673 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.747678 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:22:19.747740 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:22:19.747790 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5675,"thread_start_us":74,"threads_started":1}
I20260810 15:22:19.747906 17889 tablet_bootstrap.cc:492] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Bootstrap starting.
I20260810 15:22:19.748231 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:19.728489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.4.64.66:43731)
0810 15:22:19.734762 (+ 6273us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:19.734768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:19.734779 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:22:19.734818 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:19.734820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:19.734849 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:19.734934 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:19.735208 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.741453 (+ 6245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.743918 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:19.744726 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:19.744803 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:19.744900 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:19.748092 (+ 3192us) server_negotiation.cc:299] Negotiation successful
0810 15:22:19.748146 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6230,"thread_start_us":84,"threads_started":1}
I20260810 15:22:19.748275 17890 tablet_bootstrap.cc:654] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: Neither blocks nor log segments found. Creating new log.
I20260810 15:22:19.748781 17890 log.cc:824] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: Log is configured to *not* fsync() on all Append() calls
I20260810 15:22:19.749096 17889 tablet_bootstrap.cc:654] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Neither blocks nor log segments found. Creating new log.
I20260810 15:22:19.749464 17889 log.cc:824] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Log is configured to *not* fsync() on all Append() calls
I20260810 15:22:19.749986 17890 tablet_bootstrap.cc:492] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: No bootstrap required, opened a new log
I20260810 15:22:19.750068 17890 ts_tablet_manager.cc:1397] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:22:19.753403 17890 raft_consensus.cc:348] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:19.753651 17890 raft_consensus.cc:374] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:22:19.753520 17633 tablet_service.cc:1467] Processing CreateTablet for tablet cd80a8fdadb24880a87eefe9d4d8ca9c (DEFAULT_TABLE table=test-table [id=aff6c7a4985b4015b4b07f6ba4112129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:22:19.753707 17890 raft_consensus.cc:729] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6cfe57e5ad48c5bb2ac71e44b77bfd, State: Initialized, Role: FOLLOWER
I20260810 15:22:19.753881 17890 consensus_queue.cc:260] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:19.753957 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd80a8fdadb24880a87eefe9d4d8ca9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:22:19.754395 17890 ts_tablet_manager.cc:1428] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 15:22:19.754545 17546 heartbeater.cc:503] Master 127.4.64.126:35673 was elected leader, sending a full tablet report...
I20260810 15:22:19.757285 17895 tablet_bootstrap.cc:492] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Bootstrap starting.
I20260810 15:22:19.758452 17889 tablet_bootstrap.cc:492] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: No bootstrap required, opened a new log
I20260810 15:22:19.758571 17889 ts_tablet_manager.cc:1397] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260810 15:22:19.759019 17895 tablet_bootstrap.cc:654] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Neither blocks nor log segments found. Creating new log.
I20260810 15:22:19.759400 17895 log.cc:824] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Log is configured to *not* fsync() on all Append() calls
I20260810 15:22:19.760245 17895 tablet_bootstrap.cc:492] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: No bootstrap required, opened a new log
I20260810 15:22:19.760327 17895 ts_tablet_manager.cc:1397] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 15:22:19.761170 17889 raft_consensus.cc:348] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:19.761430 17889 raft_consensus.cc:374] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:22:19.761483 17889 raft_consensus.cc:729] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3870f1189fa64ff681c1bad727b027ee, State: Initialized, Role: FOLLOWER
I20260810 15:22:19.761729 17889 consensus_queue.cc:260] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:19.762058 17889 ts_tablet_manager.cc:1428] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:22:19.762195 17868 heartbeater.cc:503] Master 127.4.64.126:35673 was elected leader, sending a full tablet report...
I20260810 15:22:19.762686 17895 raft_consensus.cc:348] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:19.762861 17895 raft_consensus.cc:374] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:22:19.762914 17895 raft_consensus.cc:729] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db1660f7a134da0a6306ab0c9f88451, State: Initialized, Role: FOLLOWER
I20260810 15:22:19.763088 17895 consensus_queue.cc:260] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:19.763396 17895 ts_tablet_manager.cc:1428] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260810 15:22:19.763535 17709 heartbeater.cc:503] Master 127.4.64.126:35673 was elected leader, sending a full tablet report...
Aug 10 15:22:19 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
W20260810 15:22:19.789557 17547 tablet.cc:2369] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:22:19.906265 17711 tablet.cc:2369] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:22:19.939518 17869 tablet.cc:2369] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 15:22:20 dist-test-slave-9zdj krb5kdc[16743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:22:20.113812 17897 raft_consensus.cc:482] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:22:20.113993 17897 raft_consensus.cc:504] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:20.114614 17897 leader_election.cc:288] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e6cfe57e5ad48c5bb2ac71e44b77bfd (127.4.64.65:41243), 4db1660f7a134da0a6306ab0c9f88451 (127.4.64.66:43731)
I20260810 15:22:20.119371 17898 raft_consensus.cc:482] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:22:20.119519 17898 raft_consensus.cc:504] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:20.119676 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.115041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:43731 (local address 127.4.64.67:54047)
0810 15:22:20.115707 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:20.115719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:20.115796 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:20.116003 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:20.116004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:20.116005 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:20.116070 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.116071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.117028 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.117029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.118359 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.118362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.119260 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.119261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.119345 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.119350 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:22:20.119541 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 15:22:20.119587 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":8,"thread_start_us":99,"threads_started":1}
I20260810 15:22:20.120000 17898 leader_election.cc:288] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3870f1189fa64ff681c1bad727b027ee (127.4.64.67:40403), 1e6cfe57e5ad48c5bb2ac71e44b77bfd (127.4.64.65:41243)
I20260810 15:22:20.120463 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.115569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54047 (local address 127.4.64.66:43731)
0810 15:22:20.115638 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:20.115640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:20.115771 (+ 131us) server_negotiation.cc:407] Connection header received
0810 15:22:20.115879 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:20.115880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:20.115893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:20.115929 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:20.116150 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.116918 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.118483 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.119110 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.119163 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.119175 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:20.119440 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 15:22:20.119491 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:22:20.120745 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd80a8fdadb24880a87eefe9d4d8ca9c" candidate_uuid: "3870f1189fa64ff681c1bad727b027ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db1660f7a134da0a6306ab0c9f88451" is_pre_election: true
I20260810 15:22:20.120880 17650 raft_consensus.cc:2454] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3870f1189fa64ff681c1bad727b027ee in term 0.
I20260810 15:22:20.121109 17736 leader_election.cc:302] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3870f1189fa64ff681c1bad727b027ee, 4db1660f7a134da0a6306ab0c9f88451; no voters:
I20260810 15:22:20.121272 17897 raft_consensus.cc:2781] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:22:20.121332 17897 raft_consensus.cc:482] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:22:20.121333 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.115489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54927 (local address 127.4.64.65:41243)
0810 15:22:20.115545 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:20.115547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:20.115559 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:22:20.115593 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:20.115594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:20.115608 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:20.115672 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:20.115965 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.116841 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.118581 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.119212 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.119280 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.119290 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:20.121151 (+ 1861us) server_negotiation.cc:299] Negotiation successful
0810 15:22:20.121221 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:22:20.121363 17897 raft_consensus.cc:3037] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:22:20.121565 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.114913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41243 (local address 127.4.64.67:54927)
0810 15:22:20.115266 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:20.115280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:20.115336 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:20.115788 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:20.115790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:20.115792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:20.115873 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.115875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.116950 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.116951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.118485 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.118487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.120834 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.120835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.120946 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.120951 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:22:20.121017 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:22:20.121070 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":13}
I20260810 15:22:20.121919 17478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd80a8fdadb24880a87eefe9d4d8ca9c" candidate_uuid: "3870f1189fa64ff681c1bad727b027ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" is_pre_election: true
I20260810 15:22:20.122069 17478 raft_consensus.cc:2454] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3870f1189fa64ff681c1bad727b027ee in term 0.
I20260810 15:22:20.122500 17897 raft_consensus.cc:504] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:20.122637 17897 leader_election.cc:288] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [CANDIDATE]: Term 1 election: Requested vote from peers 1e6cfe57e5ad48c5bb2ac71e44b77bfd (127.4.64.65:41243), 4db1660f7a134da0a6306ab0c9f88451 (127.4.64.66:43731)
I20260810 15:22:20.122853 17478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd80a8fdadb24880a87eefe9d4d8ca9c" candidate_uuid: "3870f1189fa64ff681c1bad727b027ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd"
I20260810 15:22:20.122928 17478 raft_consensus.cc:3037] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:22:20.123886 17478 raft_consensus.cc:2454] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3870f1189fa64ff681c1bad727b027ee in term 1.
I20260810 15:22:20.124121 17736 leader_election.cc:302] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e6cfe57e5ad48c5bb2ac71e44b77bfd, 3870f1189fa64ff681c1bad727b027ee; no voters:
I20260810 15:22:20.124163 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd80a8fdadb24880a87eefe9d4d8ca9c" candidate_uuid: "3870f1189fa64ff681c1bad727b027ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db1660f7a134da0a6306ab0c9f88451"
I20260810 15:22:20.124266 17650 raft_consensus.cc:3037] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:22:20.125188 17650 raft_consensus.cc:2454] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3870f1189fa64ff681c1bad727b027ee in term 1.
I20260810 15:22:20.125705 17897 raft_consensus.cc:2781] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:22:20.125890 17897 raft_consensus.cc:686] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 1 LEADER]: Becoming Leader. State: Replica: 3870f1189fa64ff681c1bad727b027ee, State: Running, Role: LEADER
I20260810 15:22:20.125995 17897 consensus_queue.cc:237] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } }
I20260810 15:22:20.126976 17302 catalog_manager.cc:5495] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 3870f1189fa64ff681c1bad727b027ee (127.4.64.67). New cstate: current_term: 1 leader_uuid: "3870f1189fa64ff681c1bad727b027ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3870f1189fa64ff681c1bad727b027ee" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 } health_report { overall_health: UNKNOWN } } }
I20260810 15:22:20.130656 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.120303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:44799 (local address 127.4.64.67:40403)
0810 15:22:20.120381 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:20.120383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:20.120424 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:22:20.120533 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:20.120534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:20.120545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:20.120582 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:20.123342 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.124077 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.129356 (+ 5279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.130119 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.130175 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.130203 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:20.130527 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 15:22:20.130577 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 15:22:20.131116 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.120127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:40403 (local address 127.4.64.66:44799)
0810 15:22:20.120358 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:20.120368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:20.120452 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:20.123143 (+ 2691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:20.123144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:20.123147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:20.123237 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.123239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.127597 (+ 4358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.127597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.129157 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.129161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.130338 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.130339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.130425 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.130430 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:22:20.130961 (+ 531us) client_negotiation.cc:240] Negotiation successful
0810 15:22:20.131017 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260810 15:22:20.131637 17814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd80a8fdadb24880a87eefe9d4d8ca9c" candidate_uuid: "4db1660f7a134da0a6306ab0c9f88451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3870f1189fa64ff681c1bad727b027ee" is_pre_election: true
I20260810 15:22:20.135524 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.120421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:52645 (local address 127.4.64.65:41243)
0810 15:22:20.121408 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:20.121410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:20.126367 (+ 4957us) server_negotiation.cc:407] Connection header received
0810 15:22:20.126466 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:20.126467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:20.126476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:20.126505 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:22:20.126705 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.127423 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.134344 (+ 6921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.135015 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.135104 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.135118 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:20.135383 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 15:22:20.135438 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":85,"threads_started":1}
I20260810 15:22:20.135977 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.120183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41243 (local address 127.4.64.66:52645)
0810 15:22:20.126287 (+ 6104us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:20.126301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:22:20.126396 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:22:20.126562 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:22:20.126563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:22:20.126564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:22:20.126636 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.126637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.127856 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.127857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.130060 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:22:20.130063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.135161 (+ 5098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.135162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:22:20.135274 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.135279 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:22:20.135800 (+ 521us) client_negotiation.cc:240] Negotiation successful
0810 15:22:20.135859 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6087,"thread_start_us":89,"threads_started":1}
I20260810 15:22:20.136433 17478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd80a8fdadb24880a87eefe9d4d8ca9c" candidate_uuid: "4db1660f7a134da0a6306ab0c9f88451" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" is_pre_election: true
I20260810 15:22:20.136565 17478 raft_consensus.cc:2379] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4db1660f7a134da0a6306ab0c9f88451 in current term 1: Already voted for candidate 3870f1189fa64ff681c1bad727b027ee in this term.
I20260810 15:22:20.136839 17577 leader_election.cc:302] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4db1660f7a134da0a6306ab0c9f88451; no voters: 1e6cfe57e5ad48c5bb2ac71e44b77bfd, 3870f1189fa64ff681c1bad727b027ee
I20260810 15:22:20.137069 17898 raft_consensus.cc:2726] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 15:22:20.188756 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.185071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.4.64.67:40403)
0810 15:22:20.185134 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:22:20.185136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:22:20.185144 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:22:20.185190 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:22:20.185191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:22:20.185217 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:22:20.185290 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:22:20.185675 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.186447 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.187573 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:22:20.187849 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:22:20.187915 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:22:20.188126 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:22:20.188277 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:22:20.188381 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:22:20.188640 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 15:22:20.188666 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:22:20.191064 17650 raft_consensus.cc:1261] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 1 FOLLOWER]: Refusing update from remote peer 3870f1189fa64ff681c1bad727b027ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:22:20.191041 17478 raft_consensus.cc:1261] T cd80a8fdadb24880a87eefe9d4d8ca9c P 1e6cfe57e5ad48c5bb2ac71e44b77bfd [term 1 FOLLOWER]: Refusing update from remote peer 3870f1189fa64ff681c1bad727b027ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:22:20.191444 17906 consensus_queue.cc:1035] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db1660f7a134da0a6306ab0c9f88451" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:22:20.191570 17897 consensus_queue.cc:1035] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e6cfe57e5ad48c5bb2ac71e44b77bfd" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:22:20.195502 17910 mvcc.cc:205] Tried to move back new op lower bound from 7316993393419104256 to 7316993393156943872. Current Snapshot: MvccSnapshot[applied={T|T < 7316993393419104256}]
I20260810 15:22:20.198053 17913 mvcc.cc:205] Tried to move back new op lower bound from 7316993393419104256 to 7316993393156943872. Current Snapshot: MvccSnapshot[applied={T|T < 7316993393419104256}]
I20260810 15:22:20.198293 17911 mvcc.cc:205] Tried to move back new op lower bound from 7316993393419104256 to 7316993393156943872. Current Snapshot: MvccSnapshot[applied={T|T < 7316993393419104256}]
I20260810 15:22:20.198290 17302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:22:20.198398 17302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:22:20.199231 17302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86aca668106c4f03821e26522358e35f: Sending DeleteTablet for 3 replicas of tablet cd80a8fdadb24880a87eefe9d4d8ca9c
I20260810 15:22:20.199676 17476 tablet_service.cc:1514] Processing DeleteTablet for tablet cd80a8fdadb24880a87eefe9d4d8ca9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:22:20 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:47364
I20260810 15:22:20.199900 17633 tablet_service.cc:1514] Processing DeleteTablet for tablet cd80a8fdadb24880a87eefe9d4d8ca9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:22:20 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37484
I20260810 15:22:20.200241 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet cd80a8fdadb24880a87eefe9d4d8ca9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:22:20 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:45930
I20260810 15:22:20.200958 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 17395
I20260810 15:22:20.207954 17920 tablet_replica.cc:330] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: stopping tablet replica
I20260810 15:22:20.208141 17921 tablet_replica.cc:330] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: stopping tablet replica
I20260810 15:22:20.208350 17921 raft_consensus.cc:2229] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:22:20.208549 17920 raft_consensus.cc:2229] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:22:20.208760 17921 raft_consensus.cc:2258] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:22:20.208946 17920 raft_consensus.cc:2258] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:22:20.214821 17920 ts_tablet_manager.cc:1905] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:22:20.215343 17921 ts_tablet_manager.cc:1905] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:22:20.219257 17920 ts_tablet_manager.cc:1918] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:22:20.219496 17921 ts_tablet_manager.cc:1918] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:22:20.219623 17921 log.cc:1196] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/cd80a8fdadb24880a87eefe9d4d8ca9c
I20260810 15:22:20.219980 17921 ts_tablet_manager.cc:1939] T cd80a8fdadb24880a87eefe9d4d8ca9c P 3870f1189fa64ff681c1bad727b027ee: Deleting consensus metadata
I20260810 15:22:20.220139 17920 log.cc:1196] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/cd80a8fdadb24880a87eefe9d4d8ca9c
I20260810 15:22:20.220718 17286 catalog_manager.cc:4841] TS 3870f1189fa64ff681c1bad727b027ee (127.4.64.67:40403): tablet cd80a8fdadb24880a87eefe9d4d8ca9c (table test-table [id=aff6c7a4985b4015b4b07f6ba4112129]) successfully deleted
I20260810 15:22:20.221143 17920 ts_tablet_manager.cc:1939] T cd80a8fdadb24880a87eefe9d4d8ca9c P 4db1660f7a134da0a6306ab0c9f88451: Deleting consensus metadata
I20260810 15:22:20.221688 17284 catalog_manager.cc:4841] TS 4db1660f7a134da0a6306ab0c9f88451 (127.4.64.66:43731): tablet cd80a8fdadb24880a87eefe9d4d8ca9c (table test-table [id=aff6c7a4985b4015b4b07f6ba4112129]) successfully deleted
W20260810 15:22:20.221977 17284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:41243 (error 108)
I20260810 15:22:20.223026 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 17558
I20260810 15:22:20.223043 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:22:20.222886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:41243 (local address 127.0.0.1:47370)
0810 15:22:20.222941 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 15:22:20.223006 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:22:20.223556 17284 catalog_manager.cc:4584] TS 1e6cfe57e5ad48c5bb2ac71e44b77bfd (127.4.64.65:41243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd80a8fdadb24880a87eefe9d4d8ca9c: Network error: Client connection negotiation failed: client connection to 127.4.64.65:41243: connect: Connection refused (error 111)
I20260810 15:22:20.233754 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 17715
I20260810 15:22:20.241101 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 17266
I20260810 15:22:20.250012 4353 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 15:22:20.595296 4353 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 15:22:20.595424 4353 mini_ranger.cc:66] Stopping Ranger...
I20260810 15:22:20.940403 4353 mini_ranger.cc:68] Stopped Ranger
2026-08-10 15:22:20.940 UTC [16777] LOG: received smart shutdown request
2026-08-10 15:22:20.945 UTC [16777] LOG: background worker "logical replication launcher" (PID 16786) exited with exit code 1
2026-08-10 15:22:20.945 UTC [16781] LOG: shutting down
2026-08-10 15:22:21.139 UTC [16777] LOG: database system is shut down
2026-08-10T15:22:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52837 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:22:21 dist-test-slave-9zdj krb5kdc[17936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:22:21 dist-test-slave-9zdj krb5kdc[17936](info): set up 2 sockets
Aug 10 15:22:21 dist-test-slave-9zdj krb5kdc[17936](info): commencing operation
krb5kdc: starting...
W20260810 15:22:22.394083 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:22:22 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:22:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/rangeradmin_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/rangeradmin_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/rangerlookup_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/rangerlookup_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
I20260810 15:22:22.594858 4353 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task97bH11/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/postgres -l logfile start
2026-08-10 15:22:25.406 UTC [17975] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 15:22:25.406 UTC [17975] LOG: listening on IPv4 address "127.4.64.84", port 53721
2026-08-10 15:22:25.410 UTC [17975] LOG: listening on Unix socket "/tmp/.s.PGSQL.53721"
2026-08-10 15:22:25.447 UTC [17979] LOG: database system was shut down at 2026-08-10 15:22:23 UTC
2026-08-10 15:22:25.453 UTC [17975] LOG: database system is ready to accept connections
I20260810 15:22:26.528796 4353 mini_postgres.cc:96] Postgres bound to 53721
2026-08-10 15:22:26.537 UTC [17988] FATAL: database "slave" does not exist
127.4.64.84:53721 - accepting connections
I20260810 15:22:26.537971 4353 mini_ranger.cc:161] Starting Ranger...
I20260810 15:22:26.554998 4353 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 15:22:27.127508 4353 mini_ranger.cc:87] Created ranger Postgres database
I20260810 15:22:27.127610 4353 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-admin
2026-08-10 15:22:27,451 [I] DB FLAVOR :POSTGRES
2026-08-10 15:22:27,451 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:22:27,451 [I] Checking connection..
2026-08-10 15:22:27,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 15:22:27,738 [I] Checking connection passed.
2026-08-10 15:22:27,739 [I] --------- Verifying version history table ---------
2026-08-10 15:22:27,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:22:28,014 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:22:28,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:22:28,293 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:22:28,293 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 15:22:28,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 15:22:28.576 UTC [18072] WARNING: there is no transaction in progress
2026-08-10 15:22:28,584 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 15:22:28,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:22:28,883 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 15:22:28,884 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:22:28,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 15:22:29,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 15:22:29,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9zdj.c.gcp-upstream.internal','N') ;"
2026-08-10 15:22:29,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:22:29,947 [I] Table x_portal_user does not exist in database ranger
2026-08-10 15:22:29,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:22:30,203 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 15:22:30,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:22:30,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:22:30,694 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 15:22:30,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 15:22:31.771 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.789 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.809 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.827 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.846 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.866 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.929 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.936 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.952 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.963 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.973 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.988 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:31.998 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:32.006 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:32.013 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:32.021 UTC [18198] WARNING: there is no transaction in progress
2026-08-10 15:22:32,779 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 15:22:32,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 15:22:33,025 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 15:22:33,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:22:33,305 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 15:22:33,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:22:33,558 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 15:22:33,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:22:33,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:22:34,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9zdj.c.gcp-upstream.internal';"
2026-08-10 15:22:34,355 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 15:22:34,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 15:22:34,615 [I] DB_PATCHES have already been applied
I20260810 15:22:34.621374 4353 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-admin:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task97bH11/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 15:22:34.621475 4353 mini_ranger.cc:193] Using host: 127.4.64.84
I20260810 15:22:34.626245 4353 mini_ranger.cc:227] Ranger admin URL: http://127.4.64.84:53961
Aug 10, 2026 3:22:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:22:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:22:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:22:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:22:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:22:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:22:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 15:22:35.837059 4353 mini_ranger.cc:160] Time spent starting Ranger: real 9.299s user 0.001s sys 0.012s
Aug 10, 2026 3:22:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:22:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 3:22:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:22:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:22:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 15:22:57.628031 4353 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.64.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.64.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/rangerkms_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.64.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/rangerkms_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.64.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.64.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.64.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/HTTP_127.4.64.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 15:22:57 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375377, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 15:22:57.730295 4353 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 15:22:57.742578 4353 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 15:22:58.793951 4353 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 15:22:58.794042 4353 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms
I20260810 15:22:58.794067 4353 mini_ranger_kms.cc:227] Using postgres at 127.4.64.84:53721
2026-08-10 15:22:58,970 [I] DB FLAVOR :POSTGRES
2026-08-10 15:22:58,970 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:22:58,970 [I] Checking connection
2026-08-10 15:22:58,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 15:22:59,219 [I] connection success
2026-08-10 15:22:59,219 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 15:22:59,219 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 15:22:59,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 15:22:59,467 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 15:22:59,468 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:22:59,468 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 15:22:59,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.64.84:53721/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 15:22:59,741 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 15:23:00.260998 4353 mini_ranger_kms.cc:310] Created kms service
I20260810 15:23:00.460600 4353 mini_ranger_kms.cc:326] Created kudu user
I20260810 15:23:00.509378 4353 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 15:23:00 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375377, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:00.924026 4353 mini_ranger_kms.cc:384] Added ranger policy
I20260810 15:23:00.924257 4353 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms:/tmp/dist-test-task97bH11/build/release/bin/postgresql.jar:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task97bH11/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 15:23:00.924286 4353 mini_ranger_kms.cc:240] Using host: 127.4.64.84
I20260810 15:23:00.929786 4353 mini_ranger_kms.cc:276] Ranger KMS PID: 18408
I20260810 15:23:00.929876 4353 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.64.84:55215
Aug 10, 2026 3:23:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:23:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:23:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:23:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:23:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:23:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 15:23:03.427024 4353 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.697s user 0.000s sys 0.160s
I20260810 15:23:03.427157 4353 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 15:23:03.427227 4353 mini_ranger_kms.cc:401] 127.4.64.84:55215/kms/v1/keys
Aug 10 15:23:03 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.64.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 15:23:03 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.64.84@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
Aug 10 15:23:04 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.64.84@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
Aug 10 15:23:06 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:23:06.621412 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:32979
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:32979
--ranger_config_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:06.724713 18469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:06.724936 18469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:06.724959 18469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:06.726208 18469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:06.726253 18469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:06.726267 18469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:06.726280 18469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:06.726292 18469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:06.728017 18469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.64.84:55215/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:32979
--ranger_config_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:06.728276 18469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:06.728504 18469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:23:06.735153 18469 server_base.cc:1034] running on GCE node
W20260810 15:23:06.735294 18478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:06.735193 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:06.735195 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:06.736066 18469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:06.736392 18469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:06.737576 18469 hybrid_clock.cc:648] HybridClock initialized: now 1786375386737560 us; error 33 us; skew 500 ppm
Aug 10 15:23:06 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:06.744901 18469 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:06.745473 18469 webserver.cc:460] Webserver started at http://127.4.64.126:44181/ using document root <none> and password file <none>
I20260810 15:23:06.745750 18469 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:06.745795 18469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:06.745945 18469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:23:06 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:06.800148 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "d98b17c97a11421db01568a42965d8f6"
format_stamp: "Formatted at 2026-08-10 15:23:06 on dist-test-slave-9zdj"
server_key: "707715da9cb701f8a28cd47828263582"
server_key_iv: "75652d2aa9c4bc9870c9255d46a6d95f"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:06.800679 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "d98b17c97a11421db01568a42965d8f6"
format_stamp: "Formatted at 2026-08-10 15:23:06 on dist-test-slave-9zdj"
server_key: "707715da9cb701f8a28cd47828263582"
server_key_iv: "75652d2aa9c4bc9870c9255d46a6d95f"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:06.802615 18469 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260810 15:23:06.815603 18486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:06.815804 18469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:06.815925 18469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "d98b17c97a11421db01568a42965d8f6"
format_stamp: "Formatted at 2026-08-10 15:23:06 on dist-test-slave-9zdj"
server_key: "707715da9cb701f8a28cd47828263582"
server_key_iv: "75652d2aa9c4bc9870c9255d46a6d95f"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:06.816045 18469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:06.835678 18469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:06.838037 18469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:06.838219 18469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:06.843395 18469 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:32979
I20260810 15:23:06.843479 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:32979 every 8 connection(s)
I20260810 15:23:06.843865 18469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:06.844610 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:06.847043 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Bootstrap starting.
I20260810 15:23:06.847712 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:06.848029 18545 log.cc:824] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:06.848745 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: No bootstrap required, opened a new log
I20260810 15:23:06.849872 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 18469
I20260810 15:23:06.849982 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:23:06.850178 4353 external_mini_cluster.cc:1428] Setting key 5a5d3ff0b69d2bd288a6fe52020c1fa8
I20260810 15:23:06.851341 18545 raft_consensus.cc:348] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:06.851505 18545 raft_consensus.cc:374] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:06.851569 18545 raft_consensus.cc:729] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98b17c97a11421db01568a42965d8f6, State: Initialized, Role: FOLLOWER
I20260810 15:23:06.851717 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:06.851830 18545 raft_consensus.cc:388] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:06.851866 18545 raft_consensus.cc:482] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:06.851945 18545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:06.852689 18545 raft_consensus.cc:504] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:06.852823 18545 leader_election.cc:302] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d98b17c97a11421db01568a42965d8f6; no voters:
I20260810 15:23:06.853032 18545 leader_election.cc:288] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:23:06.853061 18550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:06.853324 18550 raft_consensus.cc:686] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 LEADER]: Becoming Leader. State: Replica: d98b17c97a11421db01568a42965d8f6, State: Running, Role: LEADER
I20260810 15:23:06.853413 18545 sys_catalog.cc:564] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:06.853461 18550 consensus_queue.cc:237] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:06.853863 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d98b17c97a11421db01568a42965d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } } }
I20260810 15:23:06.854031 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:06.854243 18551 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d98b17c97a11421db01568a42965d8f6. Latest consensus state: current_term: 1 leader_uuid: "d98b17c97a11421db01568a42965d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } } }
I20260810 15:23:06.854387 18551 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:06.854966 18545 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 10 15:23:06 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:06.858717 18567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:06.862546 18567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:06.864372 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:06.850758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.4.64.126:32979)
0810 15:23:06.850936 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:06.850945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:06.850978 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:23:06.851347 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:06.851357 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:06.851425 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:06.851577 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:06.852539 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:06.854095 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:06.855298 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:06.856246 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:06.856359 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:06.859525 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:06.859570 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:06.859570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:06.859648 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:06.862402 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:06.862813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:06.862816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:06.862816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:06.862957 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:06.863164 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:06.863165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:06.863165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:06.863530 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:06.863691 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:06.863830 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 15:23:06.864093 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260810 15:23:06.870918 18567 catalog_manager.cc:1347] Generated new cluster ID: b1252fb076eb4525ad264a8fb2104454
I20260810 15:23:06.870975 18567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:23:06.882563 18567 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:23:06.883735 18567 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:06.890400 18567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Generated new TSK 0
I20260810 15:23:06.890724 18567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:23:07.695653 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32979
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:07.827117 18599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:07.827397 18599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:07.827452 18599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:07.828812 18599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:07.828894 18599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:07.828946 18599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:07.830919 18599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.64.84:55215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:07.831247 18599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:07.831545 18599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:07.837541 18610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:07.837728 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:07.837834 18612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:07.838924 18599 server_base.cc:1034] running on GCE node
I20260810 15:23:07.839203 18599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:07.839495 18599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:07.842792 18599 hybrid_clock.cc:648] HybridClock initialized: now 1786375387842762 us; error 41 us; skew 500 ppm
Aug 10 15:23:07 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:07.846365 18599 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:07.846967 18599 webserver.cc:460] Webserver started at http://127.4.64.65:45675/ using document root <none> and password file <none>
I20260810 15:23:07.847306 18599 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:07.847395 18599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:07.847543 18599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:23:07 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:07.866947 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "70147f924da840fe9c7a138116198532"
format_stamp: "Formatted at 2026-08-10 15:23:07 on dist-test-slave-9zdj"
server_key: "5517e25cfee57948355f4cb491d18abb"
server_key_iv: "45e529b0fe4c9ee22c96173f5591960c"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:07.867440 18599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "70147f924da840fe9c7a138116198532"
format_stamp: "Formatted at 2026-08-10 15:23:07 on dist-test-slave-9zdj"
server_key: "5517e25cfee57948355f4cb491d18abb"
server_key_iv: "45e529b0fe4c9ee22c96173f5591960c"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:07.870018 18599 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 15:23:07.901762 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:07.902065 18599 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.001s sys 0.000s
I20260810 15:23:07.902168 18599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "70147f924da840fe9c7a138116198532"
format_stamp: "Formatted at 2026-08-10 15:23:07 on dist-test-slave-9zdj"
server_key: "5517e25cfee57948355f4cb491d18abb"
server_key_iv: "45e529b0fe4c9ee22c96173f5591960c"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:07.902275 18599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:07.931568 18599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:07.934134 18599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:07.934372 18599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:07.934679 18599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:07.935117 18599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:07.935220 18599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:07.935328 18599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:07.935405 18599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:07.944470 18599 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:40589
I20260810 15:23:07.945020 18599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:07.945247 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 18599
I20260810 15:23:07.945361 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:23:07.945583 4353 external_mini_cluster.cc:1428] Setting key 7f3dc876d4cf53621f75669ebbfba091
I20260810 15:23:07.945762 18758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:40589 every 8 connection(s)
Aug 10 15:23:07 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:07.958945 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:07.946535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32979 (local address 127.4.64.65:47117)
0810 15:23:07.947217 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:07.947273 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:07.947907 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:07.948423 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:07.948428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:07.948692 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:07.948973 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:07.948984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:07.950087 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:07.950088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:07.951448 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:07.951451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:07.952046 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:07.952047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:07.952217 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:07.952852 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:07.952863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:07.954753 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:07.956939 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:07.956947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:07.956950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:07.957323 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:07.957569 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:07.957570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:07.957570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:07.957780 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:07.958123 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:07.958130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:07.958350 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 15:23:07.958468 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 15:23:07.958666 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":88,"threads_started":1}
I20260810 15:23:07.959659 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:07.946829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:47117 (local address 127.4.64.126:32979)
0810 15:23:07.948095 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:07.948098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:07.948105 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:23:07.948130 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:07.948131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:07.948143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:07.948223 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:07.949107 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:07.949977 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:07.951554 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:07.951863 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:07.951940 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:07.954879 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:07.954881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:07.954881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:07.954890 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:07.956830 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:07.957422 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:07.957423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:07.957424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:07.957494 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:07.957865 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:07.957866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:07.957866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:07.958031 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:07.958093 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:07.959396 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0810 15:23:07.959499 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":56,"threads_started":1}
I20260810 15:23:07.960389 18761 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32979
I20260810 15:23:07.960529 18761 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:07.960728 18761 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:07.961331 18506 ts_manager.cc:194] Registered new tserver with Master: 70147f924da840fe9c7a138116198532 (127.4.64.65:40589)
I20260810 15:23:07.962316 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:47117
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:23:07.989675 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32979
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 15:23:08.107275 18767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:08.107626 18767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:08.107669 18767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:08.109813 18767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:08.109905 18767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:08.109951 18767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:08.112607 18767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.64.84:55215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:08.112951 18767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:08.113277 18767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:08.121302 18774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.121381 18773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.121963 18779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:08.131847 18767 server_base.cc:1034] running on GCE node
I20260810 15:23:08.132220 18767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:08.132601 18767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:08.134328 18767 hybrid_clock.cc:648] HybridClock initialized: now 1786375388134305 us; error 40 us; skew 500 ppm
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:08.138028 18767 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:08.138645 18767 webserver.cc:460] Webserver started at http://127.4.64.66:37697/ using document root <none> and password file <none>
I20260810 15:23:08.138959 18767 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:08.139038 18767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:08.139202 18767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:08.160687 18767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "6aa20a0dcdd5407bb4f5e41a426f8655"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "bbae14c45171cc49ede51864a2529716"
server_key_iv: "9a43572d3492ea4849c5f59300464053"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.161211 18767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "6aa20a0dcdd5407bb4f5e41a426f8655"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "bbae14c45171cc49ede51864a2529716"
server_key_iv: "9a43572d3492ea4849c5f59300464053"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.166987 18767 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 15:23:08.185240 18789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.185460 18767 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 15:23:08.185555 18767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "6aa20a0dcdd5407bb4f5e41a426f8655"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "bbae14c45171cc49ede51864a2529716"
server_key_iv: "9a43572d3492ea4849c5f59300464053"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.185700 18767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:08.202159 18767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:08.204757 18767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:08.204926 18767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:08.205165 18767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:08.205644 18767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:08.205713 18767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.205755 18767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:08.205775 18767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.216050 18767 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38961
I20260810 15:23:08.216599 18767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:23:08.217357 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 18767
I20260810 15:23:08.217478 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:23:08.217724 4353 external_mini_cluster.cc:1428] Setting key 91843eee7b5be663c7cf324e8878bd3c
I20260810 15:23:08.218129 18933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38961 every 8 connection(s)
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
I20260810 15:23:08.241707 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:08.219056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:59449 (local address 127.4.64.126:32979)
0810 15:23:08.219112 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:08.219114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:08.221700 (+ 2586us) server_negotiation.cc:407] Connection header received
0810 15:23:08.225894 (+ 4194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:08.225896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:08.225913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:08.225950 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:08.226933 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.227776 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.229393 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.229771 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.229822 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:08.237986 (+ 8164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:08.237988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:08.237989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:08.237999 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:08.239874 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:08.240473 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:08.240474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:08.240475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:08.240539 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:08.240835 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:08.240837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:08.240837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:08.240987 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:08.241029 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:08.241455 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 15:23:08.241555 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:23:08.242705 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:08.218741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32979 (local address 127.4.64.66:59449)
0810 15:23:08.221524 (+ 2783us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:08.221571 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:08.222336 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:08.226188 (+ 3852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:08.226194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:08.226492 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:08.226805 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:08.226812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.227908 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.227909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:08.229264 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:08.229267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.229941 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.229942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:08.230097 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:08.231731 (+ 1634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:08.231742 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:08.233705 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:08.239988 (+ 6283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:08.239999 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:08.240003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:08.240371 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:08.240610 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:08.240611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:08.240611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:08.240757 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:08.241075 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:08.241082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:08.241295 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 15:23:08.242270 (+ 975us) client_negotiation.cc:240] Negotiation successful
0810 15:23:08.242458 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2737,"thread_start_us":88,"threads_started":1}
I20260810 15:23:08.243430 18936 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32979
I20260810 15:23:08.243530 18936 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:08.243726 18936 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:08.244117 18506 ts_manager.cc:194] Registered new tserver with Master: 6aa20a0dcdd5407bb4f5e41a426f8655 (127.4.64.66:38961)
I20260810 15:23:08.244652 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:59449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:23:08.269372 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32979
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:08.386739 18941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:08.387070 18941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:08.387138 18941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:08.389017 18941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:08.389111 18941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:08.389153 18941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:08.391671 18941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.64.84:55215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:08.391963 18941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:08.392283 18941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:08.398322 18946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.398322 18947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.398619 18949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:08.398891 18941 server_base.cc:1034] running on GCE node
I20260810 15:23:08.399156 18941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:08.399447 18941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:08.402693 18941 hybrid_clock.cc:648] HybridClock initialized: now 1786375388402669 us; error 31 us; skew 500 ppm
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:08.406172 18941 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:08.406750 18941 webserver.cc:460] Webserver started at http://127.4.64.67:38237/ using document root <none> and password file <none>
I20260810 15:23:08.407050 18941 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:08.407112 18941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:08.407254 18941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:08.421371 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "43c9e81fecd84b5fbe6ea3336b5017a8"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "5fd16a33820106e7fb8e64982f79943e"
server_key_iv: "4c286eaa1b12f5cf422407ab336e94f2"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.421886 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "43c9e81fecd84b5fbe6ea3336b5017a8"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "5fd16a33820106e7fb8e64982f79943e"
server_key_iv: "4c286eaa1b12f5cf422407ab336e94f2"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.426069 18941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 15:23:08.441818 18958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.442023 18941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:23:08.442126 18941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "43c9e81fecd84b5fbe6ea3336b5017a8"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "5fd16a33820106e7fb8e64982f79943e"
server_key_iv: "4c286eaa1b12f5cf422407ab336e94f2"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.442243 18941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:08.456183 18941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:08.459129 18941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:08.459313 18941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:08.459580 18941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:08.459964 18941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:08.460014 18941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.460047 18941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:08.460074 18941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.467095 18941 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43475
I20260810 15:23:08.467557 18941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:23:08.470279 19088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43475 every 8 connection(s)
I20260810 15:23:08.471509 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 18941
I20260810 15:23:08.471619 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:23:08.471812 4353 external_mini_cluster.cc:1428] Setting key 75fb4019a82b2ccdd1a44eb20553be14
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:08.490099 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:08.478508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54091 (local address 127.4.64.126:32979)
0810 15:23:08.478564 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:08.478567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:08.478754 (+ 187us) server_negotiation.cc:407] Connection header received
0810 15:23:08.479446 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:08.479448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:08.479460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:08.479492 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:08.480465 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.481275 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.482938 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.483250 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.483311 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:08.486323 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:08.486325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:08.486325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:08.486335 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:08.488287 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:08.488857 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:08.488858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:08.488858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:08.488930 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:08.489267 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:08.489267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:08.489268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:08.489421 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:08.489489 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:08.489900 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 15:23:08.489989 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:23:08.490638 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:08.478367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32979 (local address 127.4.64.67:54091)
0810 15:23:08.478578 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:08.478638 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:08.479312 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:08.479727 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:08.479731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:08.480032 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:08.480343 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:08.480350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.481413 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.481415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:08.482820 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:08.482823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.483431 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.483433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:08.483591 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:08.484301 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:08.484313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:08.486207 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:08.488401 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:08.488411 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:08.488414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:08.488760 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:08.489019 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:08.489020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:08.489020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:08.489181 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:08.489500 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:08.489507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:08.489736 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 15:23:08.489867 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 15:23:08.490052 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260810 15:23:08.491178 19089 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32979
I20260810 15:23:08.491305 19089 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:08.491524 19089 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:08.491979 18506 ts_manager.cc:194] Registered new tserver with Master: 43c9e81fecd84b5fbe6ea3336b5017a8 (127.4.64.67:43475)
I20260810 15:23:08.492463 18506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:54091
I20260810 15:23:08.494097 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:23:08.494967 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 18599
I20260810 15:23:08.517576 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 18767
I20260810 15:23:08.526508 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 18941
I20260810 15:23:08.544301 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 18469
I20260810 15:23:08.554692 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:32979
--webserver_interface=127.4.64.126
--webserver_port=44181
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:32979
--ranger_config_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:08.641547 19095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:08.641815 19095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:08.641839 19095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:08.642973 19095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:08.643014 19095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:08.643028 19095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:08.643041 19095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:08.643054 19095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:08.644613 19095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:32979
--ranger_config_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:08.644838 19095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:08.645020 19095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:08.650238 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.650311 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.650328 19100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:08.650548 19095 server_base.cc:1034] running on GCE node
I20260810 15:23:08.650878 19095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:08.651177 19095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:08.652352 19095 hybrid_clock.cc:648] HybridClock initialized: now 1786375388652348 us; error 52 us; skew 500 ppm
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:08.655148 19095 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:08.655612 19095 webserver.cc:460] Webserver started at http://127.4.64.126:44181/ using document root <none> and password file <none>
I20260810 15:23:08.655849 19095 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:08.655927 19095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:08.706645 19095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:23:08.707618 19115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.707860 19095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:08.707963 19095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "d98b17c97a11421db01568a42965d8f6"
format_stamp: "Formatted at 2026-08-10 15:23:06 on dist-test-slave-9zdj"
server_key: "707715da9cb701f8a28cd47828263582"
server_key_iv: "75652d2aa9c4bc9870c9255d46a6d95f"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.708374 19095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:08.725706 19095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:08.727530 19095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:08.727713 19095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:08.732237 19095 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:32979
I20260810 15:23:08.732296 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:32979 every 8 connection(s)
I20260810 15:23:08.732717 19095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:08.734087 19173 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:23:08.734764 19173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Bootstrap starting.
I20260810 15:23:08.742341 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19095
I20260810 15:23:08.742758 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:40589
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:32979
--webserver_port=45675
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:08.743376 19173 log.cc:824] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:08.745525 19173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:08.745940 19173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Bootstrap complete.
I20260810 15:23:08.748517 19173 raft_consensus.cc:348] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:08.748765 19173 raft_consensus.cc:729] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98b17c97a11421db01568a42965d8f6, State: Initialized, Role: FOLLOWER
I20260810 15:23:08.748936 19173 consensus_queue.cc:260] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:08.749013 19173 raft_consensus.cc:388] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:08.749044 19173 raft_consensus.cc:482] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:08.749114 19173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:23:08.750258 19173 raft_consensus.cc:504] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:08.750351 19173 leader_election.cc:302] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d98b17c97a11421db01568a42965d8f6; no voters:
I20260810 15:23:08.750591 19173 leader_election.cc:288] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:23:08.750628 19178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:23:08.750741 19178 raft_consensus.cc:686] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [term 2 LEADER]: Becoming Leader. State: Replica: d98b17c97a11421db01568a42965d8f6, State: Running, Role: LEADER
I20260810 15:23:08.750887 19173 sys_catalog.cc:564] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:08.750910 19178 consensus_queue.cc:237] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } }
I20260810 15:23:08.751293 19179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d98b17c97a11421db01568a42965d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } } }
I20260810 15:23:08.751379 19179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:08.751734 19180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d98b17c97a11421db01568a42965d8f6. Latest consensus state: current_term: 2 leader_uuid: "d98b17c97a11421db01568a42965d8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d98b17c97a11421db01568a42965d8f6" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32979 } } }
I20260810 15:23:08.751822 19180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:08.751897 19184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:08.752458 19184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:08.752677 19184 catalog_manager.cc:1259] Loaded cluster ID: b1252fb076eb4525ad264a8fb2104454
I20260810 15:23:08.752767 19184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:23:08.752890 19173 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:23:08.756388 19184 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:08.757081 19184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Loaded TSK: 0
I20260810 15:23:08.757457 19184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:23:08.864586 19177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:08.864938 19177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:08.865005 19177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:08.866752 19177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:08.866868 19177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:08.866936 19177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:08.869302 19177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:40589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=45675
--tserver_master_addrs=127.4.64.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:08.869565 19177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:08.869985 19177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:08.877192 19217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.877278 19218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:08.877441 19220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:08.877504 19177 server_base.cc:1034] running on GCE node
I20260810 15:23:08.878051 19177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:08.878366 19177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:08.879724 19177 hybrid_clock.cc:648] HybridClock initialized: now 1786375388879703 us; error 35 us; skew 500 ppm
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:08.883132 19177 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:08.883728 19177 webserver.cc:460] Webserver started at http://127.4.64.65:45675/ using document root <none> and password file <none>
I20260810 15:23:08.884054 19177 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:08.884231 19177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:23:08 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:08.901361 19177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:23:08.902590 19228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.902842 19177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:23:08.902972 19177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "70147f924da840fe9c7a138116198532"
format_stamp: "Formatted at 2026-08-10 15:23:07 on dist-test-slave-9zdj"
server_key: "5517e25cfee57948355f4cb491d18abb"
server_key_iv: "45e529b0fe4c9ee22c96173f5591960c"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:08.903453 19177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:08.975412 19177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:08.977967 19177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:08.978193 19177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:08.978540 19177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:08.979017 19177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:08.979074 19177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.979106 19177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:08.979125 19177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:08.986660 19177 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:40589
I20260810 15:23:08.987211 19177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:08.987814 19365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:40589 every 8 connection(s)
I20260810 15:23:08.989226 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19177
I20260810 15:23:08.989553 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:38961
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:32979
--webserver_port=37697
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:09.007061 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:08.988067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32979 (local address 127.4.64.65:33457)
0810 15:23:08.988242 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:08.988287 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:08.988937 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:08.994618 (+ 5681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:08.994622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:08.994893 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:08.995176 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:08.995183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.997096 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.997097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:08.998456 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:08.998460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.999437 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.999438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:08.999599 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.000656 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:09.000668 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:09.002556 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:09.004961 (+ 2405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:09.004968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:09.004970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:09.005315 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:09.005655 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:09.005656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:09.005656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:09.005811 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:09.006339 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:09.006346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:09.006509 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 15:23:09.006615 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 15:23:09.006784 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260810 15:23:09.008109 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:08.988469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:33457 (local address 127.4.64.126:32979)
0810 15:23:08.993954 (+ 5485us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:08.993967 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:08.994022 (+ 55us) server_negotiation.cc:407] Connection header received
0810 15:23:08.994193 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:08.994200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:08.994269 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:08.994437 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:08.995323 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.996961 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.998559 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:08.999207 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:08.999316 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.002705 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:09.002710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:09.002711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:09.002724 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:09.004858 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.005420 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.005423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.005424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.005545 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.005897 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.005898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.005899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.006222 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:09.006308 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:09.007762 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0810 15:23:09.007903 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5416,"thread_start_us":86,"threads_started":1}
I20260810 15:23:09.009141 19368 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32979
I20260810 15:23:09.009248 19368 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.009446 19368 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
W20260810 15:23:09.010398 19368 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:32979 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:23:09.010468 19368 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.010528 19368 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:09.010959 19368 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.011035 19368 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
W20260810 15:23:09.011430 19368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:23:09.100740 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:09.101089 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:09.101133 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:09.103030 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:09.103102 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:09.103147 19372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:09.105348 19372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:38961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=37697
--tserver_master_addrs=127.4.64.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:09.105664 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:09.106031 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:09.112149 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:09.112207 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:09.112217 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:09.112780 19372 server_base.cc:1034] running on GCE node
I20260810 15:23:09.113039 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:09.113381 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:09.114557 19372 hybrid_clock.cc:648] HybridClock initialized: now 1786375389114514 us; error 56 us; skew 500 ppm
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:09.118239 19372 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:09.118770 19372 webserver.cc:460] Webserver started at http://127.4.64.66:37697/ using document root <none> and password file <none>
I20260810 15:23:09.119043 19372 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:09.119129 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:09.135553 19372 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:09.136770 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:09.137017 19372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:09.137111 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "6aa20a0dcdd5407bb4f5e41a426f8655"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "bbae14c45171cc49ede51864a2529716"
server_key_iv: "9a43572d3492ea4849c5f59300464053"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:09.137585 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:09.152451 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:09.154968 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:09.155135 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:09.155422 19372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:09.155822 19372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:09.155884 19372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:09.155920 19372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:09.155944 19372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:09.163755 19372 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38961
I20260810 15:23:09.163928 19524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38961 every 8 connection(s)
I20260810 15:23:09.164230 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:23:09.168452 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19372
I20260810 15:23:09.168798 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.64.84:55215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:43475
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:32979
--webserver_port=38237
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:09.175961 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:09.165132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32979 (local address 127.4.64.66:48379)
0810 15:23:09.165296 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:09.165340 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:09.165933 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:09.166221 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:09.166224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:09.166407 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:09.166605 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:09.166609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.167607 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.167608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:09.169013 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:09.169016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.169653 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.169654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:09.169812 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.170494 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:09.170505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:09.172284 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:09.174276 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:09.174283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:09.174286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:09.174595 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:09.174824 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:09.174825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:09.174825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:09.174951 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:09.175259 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:09.175266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:09.175437 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 15:23:09.175546 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 15:23:09.175720 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260810 15:23:09.176476 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:09.165175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:48379 (local address 127.4.64.126:32979)
0810 15:23:09.165247 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:09.165250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:09.165434 (+ 184us) server_negotiation.cc:407] Connection header received
0810 15:23:09.166045 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:09.166046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:09.166063 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:09.166151 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:09.166712 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.167486 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.169129 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.169449 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.169505 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.172397 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:09.172399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:09.172399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:09.172407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:09.174174 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.174680 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.174681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.174682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.174756 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.175023 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.175024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.175024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.175172 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:09.175232 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:09.176279 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0810 15:23:09.176380 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:23:09.177277 19525 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32979
I20260810 15:23:09.177392 19525 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.177601 19525 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
W20260810 15:23:09.178434 19525 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:32979 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:23:09.178517 19525 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.178591 19525 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:09.179003 19525 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.179086 19525 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
W20260810 15:23:09.179436 19525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:23:09.297451 19528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:09.297752 19528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:09.297783 19528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:09.299086 19528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:09.299149 19528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:09.299194 19528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:09.300814 19528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.64.84:55215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:43475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=38237
--tserver_master_addrs=127.4.64.126:32979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:09.301090 19528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:09.301391 19528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:09.307807 19535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:09.307893 19534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:09.308036 19528 server_base.cc:1034] running on GCE node
W20260810 15:23:09.308223 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:09.308432 19528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:09.308741 19528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:09.309923 19528 hybrid_clock.cc:648] HybridClock initialized: now 1786375389309906 us; error 30 us; skew 500 ppm
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:09.312845 19528 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:09.313367 19528 webserver.cc:460] Webserver started at http://127.4.64.67:38237/ using document root <none> and password file <none>
I20260810 15:23:09.313689 19528 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:09.313774 19528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:09.334893 19528 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260810 15:23:09.338083 19546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:09.338306 19528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 15:23:09.338440 19528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "43c9e81fecd84b5fbe6ea3336b5017a8"
format_stamp: "Formatted at 2026-08-10 15:23:08 on dist-test-slave-9zdj"
server_key: "5fd16a33820106e7fb8e64982f79943e"
server_key_iv: "4c286eaa1b12f5cf422407ab336e94f2"
server_key_version: "kuduclusterkey@0"
I20260810 15:23:09.338994 19528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:09.353495 19528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:09.355711 19528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:09.355894 19528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:09.356156 19528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:09.356487 19528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:09.356544 19528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:09.356588 19528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:09.356626 19528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:09.362914 19528 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43475
I20260810 15:23:09.362998 19683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43475 every 8 connection(s)
I20260810 15:23:09.363359 19528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:23:09.369925 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19528
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:09.378996 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:09.370400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.4.64.126:32979)
0810 15:23:09.370742 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:09.370744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:09.370751 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:23:09.370771 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:09.370772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:09.370790 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:09.370880 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:09.371691 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.372504 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.373651 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.373959 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.374033 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.375797 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:09.375798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:09.375799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:09.375806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:09.377571 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.378085 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.378086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.378086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.378150 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.378389 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.378389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.378390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.378520 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:09.378716 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:09.378790 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 15:23:09.378882 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":66,"threads_started":1}
I20260810 15:23:09.381451 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:09.364209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:44097 (local address 127.4.64.126:32979)
0810 15:23:09.364268 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:09.364270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:09.364561 (+ 291us) server_negotiation.cc:407] Connection header received
0810 15:23:09.365337 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:09.365338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:09.365351 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:09.365413 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:09.366240 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.367059 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.368602 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.368915 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.368961 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.374108 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:09.374110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:09.374110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:09.374118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:09.375849 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.377166 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.377167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.377168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.377231 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:09.377578 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:09.377579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:09.377579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:09.377752 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:09.377819 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:09.381270 (+ 3451us) server_negotiation.cc:299] Negotiation successful
0810 15:23:09.381360 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:23:09.381711 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:09.364221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32979 (local address 127.4.64.67:44097)
0810 15:23:09.364398 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:09.364447 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:09.365166 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:09.365519 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:09.365525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:09.365829 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:09.366132 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:09.366139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.367179 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.367180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:09.368483 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:09.368487 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:09.369075 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:09.369076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:09.369219 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:09.370790 (+ 1571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:09.370800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:09.373986 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:09.376715 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:09.376724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:09.376727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:09.377053 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:09.377309 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:09.377310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:09.377310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:09.377472 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:09.380925 (+ 3453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:09.380933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:09.381090 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 15:23:09.381234 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 15:23:09.381419 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260810 15:23:09.382239 19684 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32979
I20260810 15:23:09.382362 19684 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.382583 19684 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
W20260810 15:23:09.383232 19684 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:32979 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:23:09.383316 19684 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.383376 19684 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:09.383705 19684 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:09.383811 19684 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
W20260810 15:23:09.384160 19684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 15:23:09 dist-test-slave-9zdj krb5kdc[17936](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:10.011637 19368 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:10.011853 19368 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:10.012300 19134 ts_manager.cc:194] Registered new tserver with Master: 70147f924da840fe9c7a138116198532 (127.4.64.65:40589)
I20260810 15:23:10.013343 19134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:33457
I20260810 15:23:10.179630 19525 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:10.179818 19525 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:10.180260 19134 ts_manager.cc:194] Registered new tserver with Master: 6aa20a0dcdd5407bb4f5e41a426f8655 (127.4.64.66:38961)
I20260810 15:23:10.180816 19134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:48379
Aug 10 15:23:10 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:10.384371 19684 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:10.384573 19684 heartbeater.cc:511] Master 127.4.64.126:32979 requested a full tablet report, sending...
I20260810 15:23:10.384963 19133 ts_manager.cc:194] Registered new tserver with Master: 43c9e81fecd84b5fbe6ea3336b5017a8 (127.4.64.67:43475)
I20260810 15:23:10.385444 19133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:44097
I20260810 15:23:10.390914 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:23:10.399974 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.392065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.4.64.126:32979)
0810 15:23:10.392196 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.392198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.392342 (+ 144us) server_negotiation.cc:407] Connection header received
0810 15:23:10.392446 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.392447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.392460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.392524 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:10.393224 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.393887 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.394694 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.394942 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.395050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.396510 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:10.396512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:10.396512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:10.396520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:10.398569 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:10.398999 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:10.399001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:10.399001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:10.399075 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:10.399323 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:10.399324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:10.399324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:10.399460 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:10.399586 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.399732 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.399826 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260810 15:23:10.402393 19133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:23:10.404282 19133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:23:10.422272 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.407539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38961 (local address 127.0.0.1:49990)
0810 15:23:10.417707 (+ 10168us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.417721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:10.417801 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:10.418043 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:10.418047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:10.418049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:10.418167 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.418169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.420014 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.420014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.420901 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.420904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.421950 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.421951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.422046 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.422063 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 15:23:10.422115 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:23:10.422168 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10140,"mutex_wait_us":19,"thread_start_us":94,"threads_started":1}
I20260810 15:23:10.423350 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.408083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.4.64.66:38961)
0810 15:23:10.408252 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.408258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.417822 (+ 9564us) server_negotiation.cc:407] Connection header received
0810 15:23:10.417881 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.417885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.417904 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.418323 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:10.418415 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.419907 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.421002 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.421844 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.422780 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.422862 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.422891 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.422935 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260810 15:23:10.424223 19449 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8dad1648744447925d9c04358ea82d (DEFAULT_TABLE table=test-table [id=5930bb06ef0e4af8a95e6602ef96e6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:10.424651 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8dad1648744447925d9c04358ea82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:10.425807 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.407856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:40589 (local address 127.0.0.1:60450)
0810 15:23:10.417814 (+ 9958us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.417828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:10.417900 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:10.418627 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:10.418627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:10.418628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:10.418653 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.418653 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.420008 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.420009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.424466 (+ 4457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.424469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.425528 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.425529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.425593 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.425598 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:23:10.425669 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:23:10.425713 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9914,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260810 15:23:10.426101 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.407946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.4.64.65:40589)
0810 15:23:10.409744 (+ 1798us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.409748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.417978 (+ 8230us) server_negotiation.cc:407] Connection header received
0810 15:23:10.418021 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.418024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.418042 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.418111 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:10.418730 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.419907 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.424568 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.425358 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.425433 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.425492 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.425969 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.426018 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":51,"threads_started":1}
I20260810 15:23:10.426820 19267 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8dad1648744447925d9c04358ea82d (DEFAULT_TABLE table=test-table [id=5930bb06ef0e4af8a95e6602ef96e6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:10.427202 19267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8dad1648744447925d9c04358ea82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:10.427641 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.407539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43475 (local address 127.0.0.1:37928)
0810 15:23:10.409815 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.409852 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:10.409923 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:10.418515 (+ 8592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:10.418516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:10.418518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:10.418557 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.418558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.422894 (+ 4336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.422895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.423953 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.423956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.427387 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.427388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.427462 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.427468 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:23:10.427523 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:23:10.427570 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2248,"thread_start_us":108,"threads_started":1}
I20260810 15:23:10.427982 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.407850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.4.64.67:43475)
0810 15:23:10.418264 (+ 10414us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.418269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.418280 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:23:10.418314 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.418316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.418331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.418411 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:10.421688 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.422799 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.426386 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.427210 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.427289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.427344 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.427854 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.427902 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10371,"thread_start_us":79,"threads_started":1}
I20260810 15:23:10.428689 19598 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8dad1648744447925d9c04358ea82d (DEFAULT_TABLE table=test-table [id=5930bb06ef0e4af8a95e6602ef96e6ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:10.429049 19598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8dad1648744447925d9c04358ea82d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:10.431007 19721 tablet_bootstrap.cc:492] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: Bootstrap starting.
I20260810 15:23:10.431751 19721 tablet_bootstrap.cc:654] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:10.432150 19721 log.cc:824] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:10.433040 19721 tablet_bootstrap.cc:492] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: No bootstrap required, opened a new log
I20260810 15:23:10.433110 19721 ts_tablet_manager.cc:1397] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:10.433288 19720 tablet_bootstrap.cc:492] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Bootstrap starting.
I20260810 15:23:10.434022 19720 tablet_bootstrap.cc:654] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:10.434340 19720 log.cc:824] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:10.434772 19724 tablet_bootstrap.cc:492] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Bootstrap starting.
I20260810 15:23:10.435304 19721 raft_consensus.cc:348] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.435480 19721 raft_consensus.cc:374] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:10.435515 19721 raft_consensus.cc:729] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70147f924da840fe9c7a138116198532, State: Initialized, Role: FOLLOWER
I20260810 15:23:10.435627 19721 consensus_queue.cc:260] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.435742 19724 tablet_bootstrap.cc:654] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:10.435883 19721 ts_tablet_manager.cc:1428] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:23:10.436071 19724 log.cc:824] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:10.436378 19368 heartbeater.cc:503] Master 127.4.64.126:32979 was elected leader, sending a full tablet report...
I20260810 15:23:10.436723 19720 tablet_bootstrap.cc:492] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: No bootstrap required, opened a new log
I20260810 15:23:10.436815 19720 ts_tablet_manager.cc:1397] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:23:10.438436 19724 tablet_bootstrap.cc:492] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: No bootstrap required, opened a new log
I20260810 15:23:10.438556 19724 ts_tablet_manager.cc:1397] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 15:23:10.439512 19720 raft_consensus.cc:348] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.439708 19720 raft_consensus.cc:374] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:10.439764 19720 raft_consensus.cc:729] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa20a0dcdd5407bb4f5e41a426f8655, State: Initialized, Role: FOLLOWER
I20260810 15:23:10.439932 19720 consensus_queue.cc:260] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.440305 19720 ts_tablet_manager.cc:1428] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 15:23:10.440475 19525 heartbeater.cc:503] Master 127.4.64.126:32979 was elected leader, sending a full tablet report...
I20260810 15:23:10.441183 19724 raft_consensus.cc:348] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.441406 19724 raft_consensus.cc:374] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:10.441481 19724 raft_consensus.cc:729] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c9e81fecd84b5fbe6ea3336b5017a8, State: Initialized, Role: FOLLOWER
I20260810 15:23:10.441684 19724 consensus_queue.cc:260] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.442075 19724 ts_tablet_manager.cc:1428] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:23:10.442162 19684 heartbeater.cc:503] Master 127.4.64.126:32979 was elected leader, sending a full tablet report...
W20260810 15:23:10.494822 19369 tablet.cc:2369] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 15:23:10 dist-test-slave-9zdj krb5kdc[17936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for HTTP/127.4.64.84@KRBTEST.COM
I20260810 15:23:10.592353 19730 raft_consensus.cc:482] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:10.592511 19730 raft_consensus.cc:504] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.592934 19730 leader_election.cc:288] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aa20a0dcdd5407bb4f5e41a426f8655 (127.4.64.66:38961), 70147f924da840fe9c7a138116198532 (127.4.64.65:40589)
I20260810 15:23:10.601590 19719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.593767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:36917 (local address 127.4.64.66:38961)
0810 15:23:10.593831 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.593834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.593890 (+ 56us) server_negotiation.cc:407] Connection header received
0810 15:23:10.593995 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.593997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.594008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.594043 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:10.594927 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.595629 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.597694 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.598309 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.598360 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.598372 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.601448 (+ 3076us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.601506 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:23:10.602157 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.593208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38961 (local address 127.4.64.67:36917)
0810 15:23:10.593820 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.593831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:10.593921 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:10.594720 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:10.594722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:10.594727 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:10.594822 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.594825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.595851 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.595853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.597085 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.597088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.599053 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.599054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.599135 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.599140 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:23:10.601991 (+ 2851us) client_negotiation.cc:240] Negotiation successful
0810 15:23:10.602044 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":111,"threads_started":1}
I20260810 15:23:10.602759 19459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8dad1648744447925d9c04358ea82d" candidate_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" is_pre_election: true
I20260810 15:23:10.602957 19459 raft_consensus.cc:2454] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c9e81fecd84b5fbe6ea3336b5017a8 in term 0.
I20260810 15:23:10.603243 19549 leader_election.cc:302] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c9e81fecd84b5fbe6ea3336b5017a8, 6aa20a0dcdd5407bb4f5e41a426f8655; no voters:
I20260810 15:23:10.603407 19730 raft_consensus.cc:2781] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:23:10.603451 19730 raft_consensus.cc:482] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:23:10.603473 19730 raft_consensus.cc:3037] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:10.604416 19730 raft_consensus.cc:504] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.604547 19730 leader_election.cc:288] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [CANDIDATE]: Term 1 election: Requested vote from peers 6aa20a0dcdd5407bb4f5e41a426f8655 (127.4.64.66:38961), 70147f924da840fe9c7a138116198532 (127.4.64.65:40589)
I20260810 15:23:10.604979 19459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8dad1648744447925d9c04358ea82d" candidate_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655"
I20260810 15:23:10.605063 19459 raft_consensus.cc:3037] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:10.606019 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.593856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:55271 (local address 127.4.64.65:40589)
0810 15:23:10.593899 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.593901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.597287 (+ 3386us) server_negotiation.cc:407] Connection header received
0810 15:23:10.597394 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.597395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.597408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.597443 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:10.599338 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.600068 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.602696 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.603264 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.603310 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.603321 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.605863 (+ 2542us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.605924 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:23:10.606040 19459 raft_consensus.cc:2454] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c9e81fecd84b5fbe6ea3336b5017a8 in term 1.
I20260810 15:23:10.606247 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.593421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:40589 (local address 127.4.64.67:55271)
0810 15:23:10.597219 (+ 3798us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.597231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:10.597321 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:10.597504 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:10.597505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:10.597507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:10.597571 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.597572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.600178 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.600179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.601331 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:10.601334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.605685 (+ 4351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.605686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:10.605765 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.605769 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:23:10.606122 (+ 353us) client_negotiation.cc:240] Negotiation successful
0810 15:23:10.606172 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3774,"thread_start_us":69,"threads_started":1}
I20260810 15:23:10.606794 19549 leader_election.cc:302] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c9e81fecd84b5fbe6ea3336b5017a8, 6aa20a0dcdd5407bb4f5e41a426f8655; no voters:
I20260810 15:23:10.607139 19310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8dad1648744447925d9c04358ea82d" candidate_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70147f924da840fe9c7a138116198532" is_pre_election: true
I20260810 15:23:10.607303 19310 raft_consensus.cc:2454] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43c9e81fecd84b5fbe6ea3336b5017a8 in term 0.
I20260810 15:23:10.607411 19308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8dad1648744447925d9c04358ea82d" candidate_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70147f924da840fe9c7a138116198532"
I20260810 15:23:10.607462 19308 raft_consensus.cc:3037] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:10.608317 19308 raft_consensus.cc:2454] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43c9e81fecd84b5fbe6ea3336b5017a8 in term 1.
I20260810 15:23:10.608646 19730 raft_consensus.cc:2781] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:10.608829 19730 raft_consensus.cc:686] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 1 LEADER]: Becoming Leader. State: Replica: 43c9e81fecd84b5fbe6ea3336b5017a8, State: Running, Role: LEADER
I20260810 15:23:10.608925 19730 consensus_queue.cc:237] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } }
I20260810 15:23:10.609843 19133 catalog_manager.cc:5495] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43c9e81fecd84b5fbe6ea3336b5017a8 (127.4.64.67). New cstate: current_term: 1 leader_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c9e81fecd84b5fbe6ea3336b5017a8" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 } health_report { overall_health: UNKNOWN } } }
W20260810 15:23:10.616796 19685 tablet.cc:2369] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:10.645474 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.638052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.4.64.67:43475)
0810 15:23:10.641746 (+ 3694us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:10.641748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:10.641939 (+ 191us) server_negotiation.cc:407] Connection header received
0810 15:23:10.641990 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:10.641991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:10.642020 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:10.642087 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:10.642337 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.643063 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.644229 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:10.644504 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:10.644574 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:10.644805 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:10.644958 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:23:10.645047 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:10.645363 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 15:23:10.645385 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3657}
I20260810 15:23:10.663019 19308 raft_consensus.cc:1261] T 5b8dad1648744447925d9c04358ea82d P 70147f924da840fe9c7a138116198532 [term 1 FOLLOWER]: Refusing update from remote peer 43c9e81fecd84b5fbe6ea3336b5017a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:10.663419 19459 raft_consensus.cc:1261] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 1 FOLLOWER]: Refusing update from remote peer 43c9e81fecd84b5fbe6ea3336b5017a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:10.664052 19735 consensus_queue.cc:1035] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70147f924da840fe9c7a138116198532" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:10.664244 19735 consensus_queue.cc:1035] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aa20a0dcdd5407bb4f5e41a426f8655" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:23:10.666742 19526 tablet.cc:2369] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:10.676793 19738 mvcc.cc:205] Tried to move back new op lower bound from 7316993600151269376 to 7316993599934763008. Current Snapshot: MvccSnapshot[applied={T|T < 7316993600151269376}]
I20260810 15:23:10.677013 19741 mvcc.cc:205] Tried to move back new op lower bound from 7316993600151269376 to 7316993599934763008. Current Snapshot: MvccSnapshot[applied={T|T < 7316993600151269376}]
I20260810 15:23:10.677290 19739 mvcc.cc:205] Tried to move back new op lower bound from 7316993600151269376 to 7316993599934763008. Current Snapshot: MvccSnapshot[applied={T|T < 7316993600151269376}]
I20260810 15:23:10.682050 19133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:10.682168 19133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:10.682899 19133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d98b17c97a11421db01568a42965d8f6: Sending DeleteTablet for 3 replicas of tablet 5b8dad1648744447925d9c04358ea82d
I20260810 15:23:10.683421 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8dad1648744447925d9c04358ea82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:10 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:49990
I20260810 15:23:10.683691 19267 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8dad1648744447925d9c04358ea82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:10 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:60450
I20260810 15:23:10.684330 19598 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8dad1648744447925d9c04358ea82d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:10 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37928
I20260810 15:23:10.684389 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19177
I20260810 15:23:10.692050 19753 tablet_replica.cc:330] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: stopping tablet replica
I20260810 15:23:10.692236 19753 raft_consensus.cc:2229] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:10.692435 19753 raft_consensus.cc:2258] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:10.692663 19755 tablet_replica.cc:330] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: stopping tablet replica
I20260810 15:23:10.692981 19755 raft_consensus.cc:2229] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:23:10.693150 19755 raft_consensus.cc:2258] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:10.693547 19753 ts_tablet_manager.cc:1905] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:10.696087 19755 ts_tablet_manager.cc:1905] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:10.697974 19755 ts_tablet_manager.cc:1918] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:10.698058 19755 log.cc:1196] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/5b8dad1648744447925d9c04358ea82d
W20260810 15:23:10.704660 19120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:40589 (error 108)
I20260810 15:23:10.705772 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19372
I20260810 15:23:10.705777 19753 ts_tablet_manager.cc:1918] T 5b8dad1648744447925d9c04358ea82d P 6aa20a0dcdd5407bb4f5e41a426f8655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:10.717706 19755 ts_tablet_manager.cc:1939] T 5b8dad1648744447925d9c04358ea82d P 43c9e81fecd84b5fbe6ea3336b5017a8: Deleting consensus metadata
I20260810 15:23:10.718272 19116 catalog_manager.cc:4841] TS 43c9e81fecd84b5fbe6ea3336b5017a8 (127.4.64.67:43475): tablet 5b8dad1648744447925d9c04358ea82d (table test-table [id=5930bb06ef0e4af8a95e6602ef96e6ed]) successfully deleted
I20260810 15:23:10.718906 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.718648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:40589 (local address 127.0.0.1:60458)
0810 15:23:10.718820 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.718861 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260810 15:23:10.719180 19120 catalog_manager.cc:4584] TS 70147f924da840fe9c7a138116198532 (127.4.64.65:40589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b8dad1648744447925d9c04358ea82d: Network error: Client connection negotiation failed: client connection to 127.4.64.65:40589: connect: Connection refused (error 111)
I20260810 15:23:10.721505 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.721396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38961 (local address 127.0.0.1:50006)
0810 15:23:10.721445 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.721474 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.66:38961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:23:10.729777 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19528
I20260810 15:23:10.765264 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.765090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38961 (local address 127.0.0.1:50016)
0810 15:23:10.765177 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.765214 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.66:38961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 15:23:10.765794 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.765687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38961 (local address 127.0.0.1:50020)
0810 15:23:10.765727 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.765754 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.66:38961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 15:23:10.772450 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.772301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:40589 (local address 127.0.0.1:60470)
0810 15:23:10.772358 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.772406 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:23:10.772930 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:10.772848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:40589 (local address 127.0.0.1:60482)
0810 15:23:10.772882 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:10.772904 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:40589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 15:23:10.781426 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19095
I20260810 15:23:10.790397 4353 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 15:23:11.136277 4353 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 15:23:11.136389 4353 mini_ranger.cc:66] Stopping Ranger...
I20260810 15:23:11.482280 4353 mini_ranger.cc:68] Stopped Ranger
2026-08-10 15:23:11.482 UTC [17975] LOG: received smart shutdown request
2026-08-10 15:23:11.487 UTC [17975] LOG: background worker "logical replication launcher" (PID 17986) exited with exit code 1
2026-08-10 15:23:11.488 UTC [17981] LOG: shutting down
2026-08-10 15:23:11.691 UTC [17975] LOG: database system is shut down
2026-08-10T15:23:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50539 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:23:11 dist-test-slave-9zdj krb5kdc[19766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:23:11 dist-test-slave-9zdj krb5kdc[19766](info): set up 2 sockets
Aug 10 15:23:11 dist-test-slave-9zdj krb5kdc[19766](info): commencing operation
krb5kdc: starting...
W20260810 15:23:12.937891 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:23:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:23:13.081354 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44205
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44205,127.4.64.125:43573,127.4.64.124:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:13.169761 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:13.170018 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:13.170061 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:13.171363 19784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:13.171442 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:13.171468 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:13.171489 19784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:13.171517 19784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:13.173167 19784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44205,127.4.64.125:43573,127.4.64.124:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:13.173477 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:13.173790 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:13.179126 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:13.179127 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:13.179112 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:13.179534 19784 server_base.cc:1034] running on GCE node
I20260810 15:23:13.179787 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:13.180059 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:13.181228 19784 hybrid_clock.cc:648] HybridClock initialized: now 1786375393181212 us; error 29 us; skew 500 ppm
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:13.183949 19784 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:13.184463 19784 webserver.cc:460] Webserver started at http://127.4.64.126:43835/ using document root <none> and password file <none>
I20260810 15:23:13.184669 19784 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:13.184713 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:13.184809 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:13.185982 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "f936889662694926b0ecf941b9eb542e"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "d961d7f555d010d5d9b53ee09dd45b1e"
server_key_iv: "1a54c0212ac89fb5ef7d5c7cc53422e1"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.186329 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "f936889662694926b0ecf941b9eb542e"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "d961d7f555d010d5d9b53ee09dd45b1e"
server_key_iv: "1a54c0212ac89fb5ef7d5c7cc53422e1"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.187814 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:23:13.188884 19800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:13.189131 19784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:13.189380 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "f936889662694926b0ecf941b9eb542e"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "d961d7f555d010d5d9b53ee09dd45b1e"
server_key_iv: "1a54c0212ac89fb5ef7d5c7cc53422e1"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.189460 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:13.204942 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:13.207861 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:13.208009 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:13.213286 19784 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44205
I20260810 15:23:13.213300 19866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44205 every 8 connection(s)
I20260810 15:23:13.213871 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:13.214982 19867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:13.216781 19867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:23:13.218405 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19784
I20260810 15:23:13.218547 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:23:13.218809 4353 external_mini_cluster.cc:1428] Setting key f34bfddf7ffa3afff39f14cab7fe7134
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.125@KRBTEST.COM; defaulting to no policy
I20260810 15:23:13.229110 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.217535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44205 (local address 127.0.0.1:35078)
0810 15:23:13.217846 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.217900 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.218404 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.218814 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.218818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.219261 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.219519 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.219527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.220596 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.220598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.221459 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.221462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.222223 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.222224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.222303 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.222924 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.222933 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.224746 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.227266 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.227273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.227275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.227491 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.227783 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.227784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.227784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.227876 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.228428 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.228436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.228500 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.228601 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.228894 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260810 15:23:13.229132 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.217679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.4.64.126:44205)
0810 15:23:13.217858 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.217868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.218000 (+ 132us) server_negotiation.cc:407] Connection header received
0810 15:23:13.218572 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.218575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.218624 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.218724 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.219648 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.220486 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.221554 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.222097 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.222270 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.224850 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.224860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.224861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.224870 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.227119 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.227587 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.227591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.227591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.227705 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.227952 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.227953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.227954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.228325 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.228423 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.228655 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.229015 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":227,"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260810 15:23:13.230139 19867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:23:13.230527 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.230386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39112)
0810 15:23:13.230448 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.230504 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 15:23:13.230705 19803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
I20260810 15:23:13.231299 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.231132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39120)
0810 15:23:13.231165 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.231271 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260810 15:23:13.231492 19867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:43573: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab.
Entry for principal kudu/127.4.64.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab.
I20260810 15:23:13.253037 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:43573
--webserver_interface=127.4.64.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab
--principal=kudu/127.4.64.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44205,127.4.64.125:43573,127.4.64.124:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:13.280771 19867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } attempt: 1
I20260810 15:23:13.281376 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.281215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39124)
0810 15:23:13.281294 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.281344 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 15:23:13.281883 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.281739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39128)
0810 15:23:13.281805 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.281852 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 15:23:13.282032 19867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:43573: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
W20260810 15:23:13.340906 19874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:13.341130 19874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:13.341156 19874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:13.342403 19874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:13.342451 19874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:13.342466 19874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:13.342480 19874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:13.342492 19874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:13.344102 19874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44205,127.4.64.125:43573,127.4.64.124:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab
--principal=kudu/127.4.64.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.64.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:13.344430 19874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:13.344697 19874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:23:13.350306 19867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } attempt: 2
W20260810 15:23:13.350394 19879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:13.350402 19880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:13.350809 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.350625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39134)
0810 15:23:13.350703 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.350746 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 15:23:13.350401 19882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:13.350732 19874 server_base.cc:1034] running on GCE node
I20260810 15:23:13.351229 19874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:13.351311 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.351204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39138)
0810 15:23:13.351250 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.351277 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 15:23:13.351430 19867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:43573: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43573: connect: Connection refused (error 111)
I20260810 15:23:13.351642 19874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:13.352838 19874 hybrid_clock.cc:648] HybridClock initialized: now 1786375393352824 us; error 26 us; skew 500 ppm
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:13.355994 19874 init.cc:377] Logged in from keytab as kudu/127.4.64.125@KRBTEST.COM (short username kudu)
I20260810 15:23:13.356561 19874 webserver.cc:460] Webserver started at http://127.4.64.125:34745/ using document root <none> and password file <none>
I20260810 15:23:13.356897 19874 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:13.356998 19874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:13.357204 19874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:13.358573 19874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data/instance:
uuid: "d97061cafc584dd5a1c6d2b338b9bf17"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "2c7ad44dde687d00a3b1b7f521085c71"
server_key_iv: "ceb8fd05fc84de67970b3f9027bcc952"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.358996 19874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal/instance:
uuid: "d97061cafc584dd5a1c6d2b338b9bf17"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "2c7ad44dde687d00a3b1b7f521085c71"
server_key_iv: "ceb8fd05fc84de67970b3f9027bcc952"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.360553 19874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:23:13.361467 19891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:13.361708 19874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:13.361846 19874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal
uuid: "d97061cafc584dd5a1c6d2b338b9bf17"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "2c7ad44dde687d00a3b1b7f521085c71"
server_key_iv: "ceb8fd05fc84de67970b3f9027bcc952"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.361941 19874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:13.381765 19874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:13.384159 19874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:13.384359 19874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:13.389281 19874 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.125:43573
I20260810 15:23:13.389314 19957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.125:43573 every 8 connection(s)
I20260810 15:23:13.389866 19874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
I20260810 15:23:13.390919 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:13.392683 19958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:13.400229 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19874
I20260810 15:23:13.400374 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-1/wal/instance
I20260810 15:23:13.400626 4353 external_mini_cluster.cc:1428] Setting key 0650fe67f442572a899b9ddf0b22765b
I20260810 15:23:13.403072 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.393502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.4.64.126:44205)
0810 15:23:13.393587 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.393590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.393872 (+ 282us) server_negotiation.cc:407] Connection header received
0810 15:23:13.394412 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.394413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.394427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.394493 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.395103 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.395669 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.396800 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.397032 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.397097 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.399636 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.399637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.399638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.399645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.401404 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.401927 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.401928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.401928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.401996 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.402366 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.402367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.402367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.402495 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.402570 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.402887 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.402978 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 15:23:13.403290 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.393514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44205 (local address 127.0.0.1:35088)
0810 15:23:13.393746 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.393784 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.394313 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.394546 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.394549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.394817 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.395021 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.395025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.395785 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.395786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.396689 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.396691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.397216 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.397217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.397337 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.398011 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.398020 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.399522 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.401503 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.401512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.401515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.401838 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.402069 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.402070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.402071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.402280 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.402572 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.402581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.402744 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.402850 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.403019 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":149,"threads_started":1}
I20260810 15:23:13.403867 19958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.124@KRBTEST.COM; defaulting to no policy
I20260810 15:23:13.412389 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.404169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39154)
0810 15:23:13.404222 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.404232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.404270 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.404651 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.404652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.404772 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.404829 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.404830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.406289 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.406290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.406791 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.406792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.407527 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.407528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.407589 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.408049 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.408053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.409346 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.411350 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.411351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.411352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.411558 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.411797 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.411798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.411799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.411870 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.412116 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.412117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.412157 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.412217 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.412306 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:23:13.412503 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.404293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.4.64.125:43573)
0810 15:23:13.404466 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.404472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.404497 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:23:13.404523 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.404525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.404540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.404629 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.404905 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.406168 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.406883 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.407435 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.407522 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.409434 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.409443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.409444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.409451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.411248 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.411671 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.411674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.411674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.411728 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.411929 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.411930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.411930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.412062 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.412120 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.412241 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.412356 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260810 15:23:13.413167 19958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.124@KRBTEST.COM" created.
I20260810 15:23:13.413687 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.413507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46116)
0810 15:23:13.413580 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.413662 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 15:23:13.413862 19895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
I20260810 15:23:13.414394 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.414284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46126)
0810 15:23:13.414344 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.414371 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 15:23:13.414506 19958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:43281: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab.
Entry for principal kudu/127.4.64.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab.
I20260810 15:23:13.433602 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:43281
--webserver_interface=127.4.64.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab
--principal=kudu/127.4.64.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44205,127.4.64.125:43573,127.4.64.124:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:13.442637 19867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } attempt: 3
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:23:13.450472 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.443144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.4.64.125:43573)
0810 15:23:13.443239 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.443241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.443251 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:23:13.443285 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.443286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.443298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.443360 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.443727 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.444278 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.444986 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.445227 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.445313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.447691 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.447693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.447693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.447702 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.449229 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.449597 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.449598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.449599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.449689 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.449903 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.449904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.449904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.450029 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.450108 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.450282 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.450366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 15:23:13.450472 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.443046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39160)
0810 15:23:13.443116 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.443129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.443185 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.443381 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.443382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.443556 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.443635 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.443637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.444380 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.444381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.444896 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.444898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.445312 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.445313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.445402 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.445966 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.445971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.447590 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.449340 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.449341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.449342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.449522 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.449769 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.449770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.449770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.449837 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.450100 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.450101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.450153 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.450253 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.450366 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 15:23:13.450894 19867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:23:13.451275 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.451156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46138)
0810 15:23:13.451217 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.451245 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:23:13.451611 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.451550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46152)
0810 15:23:13.451580 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.451600 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 15:23:13.451705 19867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:43281: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
I20260810 15:23:13.463789 19958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } attempt: 1
I20260810 15:23:13.464383 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.464197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46162)
0810 15:23:13.464306 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.464343 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 15:23:13.464865 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.464754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46178)
0810 15:23:13.464816 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.464846 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 15:23:13.465026 19958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:43281: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
I20260810 15:23:13.482878 19867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } attempt: 1
I20260810 15:23:13.483479 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.483308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46182)
0810 15:23:13.483385 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.483439 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 15:23:13.483889 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.483784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46196)
0810 15:23:13.483837 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.483869 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 15:23:13.483989 19867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:43281: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
W20260810 15:23:13.523622 19965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:13.523905 19965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:13.523947 19965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:13.525147 19965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:13.525224 19965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:13.525249 19965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:13.525269 19965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:13.525301 19965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:13.526932 19965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44205,127.4.64.125:43573,127.4.64.124:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab
--principal=kudu/127.4.64.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.64.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:13.527247 19965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:13.527500 19965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:13.532912 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:13.532856 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:13.532853 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:13.532891 19965 server_base.cc:1034] running on GCE node
I20260810 15:23:13.533157 19958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } attempt: 2
I20260810 15:23:13.533452 19965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:13.533567 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.533431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46206)
0810 15:23:13.533497 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.533530 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 15:23:13.533818 19965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:13.533972 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.533877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46222)
0810 15:23:13.533920 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.533944 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:23:13.534073 19958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:43281: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
I20260810 15:23:13.535030 19965 hybrid_clock.cc:648] HybridClock initialized: now 1786375393535003 us; error 34 us; skew 500 ppm
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:13.537727 19965 init.cc:377] Logged in from keytab as kudu/127.4.64.124@KRBTEST.COM (short username kudu)
I20260810 15:23:13.538208 19965 webserver.cc:460] Webserver started at http://127.4.64.124:45555/ using document root <none> and password file <none>
I20260810 15:23:13.538473 19965 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:13.538547 19965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:13.538710 19965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:13.539970 19965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data/instance:
uuid: "b7ea8964523c47749cfb5392255ab327"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "e769d36d5daee1e2a184d7182b391283"
server_key_iv: "46a391b67e46e6a0208dfda926150d01"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.540349 19965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal/instance:
uuid: "b7ea8964523c47749cfb5392255ab327"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "e769d36d5daee1e2a184d7182b391283"
server_key_iv: "46a391b67e46e6a0208dfda926150d01"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.541823 19965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:23:13.542598 19987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:13.542789 19965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:13.542901 19965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal
uuid: "b7ea8964523c47749cfb5392255ab327"
format_stamp: "Formatted at 2026-08-10 15:23:13 on dist-test-slave-9zdj"
server_key: "e769d36d5daee1e2a184d7182b391283"
server_key_iv: "46a391b67e46e6a0208dfda926150d01"
server_key_version: "encryptionkey@0"
I20260810 15:23:13.542980 19965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:13.559156 19867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } attempt: 2
I20260810 15:23:13.559725 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.559550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46226)
0810 15:23:13.559624 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.559673 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 15:23:13.560163 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.560048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46242)
0810 15:23:13.560101 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.560141 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 15:23:13.560257 19867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:43281: Network error: Client connection negotiation failed: client connection to 127.4.64.124:43281: connect: Connection refused (error 111)
I20260810 15:23:13.572739 19965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:13.575232 19965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:13.575459 19965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:13.580060 19965 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.124:43281
I20260810 15:23:13.580101 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.124:43281 every 8 connection(s)
I20260810 15:23:13.580546 19965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
I20260810 15:23:13.581590 20052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:13.583361 20052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:13.590674 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 19965
I20260810 15:23:13.590844 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786375228510355-4353-0/minicluster-data/master-2/wal/instance
I20260810 15:23:13.591073 4353 external_mini_cluster.cc:1428] Setting key cd43f9477784cbc88baefd32011338a9
I20260810 15:23:13.594507 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.584289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.4.64.126:44205)
0810 15:23:13.584415 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.584418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.584519 (+ 101us) server_negotiation.cc:407] Connection header received
0810 15:23:13.585047 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.585048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.585059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.585131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.585942 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.586544 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.587690 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.587904 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.587968 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.590692 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.590693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.590694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.590703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.592730 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.593330 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.593332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.593332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.593408 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.593769 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.593770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.593770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.593883 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.593941 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.594312 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.594406 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 15:23:13.594678 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.584129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44205 (local address 127.0.0.1:35094)
0810 15:23:13.584392 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.584432 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.584894 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.585291 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.585295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.585556 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.585838 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.585843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.586659 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.586660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.587568 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.587571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.588007 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.588008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.588134 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.588835 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.588844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.590561 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.592833 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.592841 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.592844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.593187 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.593485 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.593486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.593486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.593685 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.593955 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.593961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.594140 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.594258 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.594429 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":151,"threads_started":1}
I20260810 15:23:13.595366 20052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:23:13.601992 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.591646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.4.64.124:43281)
0810 15:23:13.591854 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.591859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.592066 (+ 207us) server_negotiation.cc:407] Connection header received
0810 15:23:13.592101 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.592104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.592124 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.592225 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.592892 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.594158 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.595346 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.595867 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.595963 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.598758 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.598766 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.598766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.598772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.600467 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.600939 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.600942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.600942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.601003 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.601239 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.601240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.601240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.601394 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.601509 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.601702 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.601857 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260810 15:23:13.602914 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.595660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43573 (local address 127.0.0.1:39164)
0810 15:23:13.595718 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.595730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.595768 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.595900 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.595901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.596047 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.596094 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.596096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.596897 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.596899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.597407 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.597408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.597880 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.597881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.597971 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.598438 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.598443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.599699 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.601695 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.601696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.601697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.601970 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.602283 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.602284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.602284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.602340 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.602656 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.602656 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.602703 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.602783 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.602857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:23:13.603042 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.595754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39164 (local address 127.4.64.125:43573)
0810 15:23:13.595803 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.595805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.595813 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:23:13.595838 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.595839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.595847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.595898 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.596181 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.596794 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.597499 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.597799 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.597863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.599781 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.599783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.599784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.599792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.601560 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.602071 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.602072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.602072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.602151 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.602395 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.602396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.602397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.602554 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.602638 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.602818 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.602914 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:23:13.603404 20052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:13.611215 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.603692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46264)
0810 15:23:13.603753 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.603764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.603802 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.603936 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.603937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.604057 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.604090 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.604091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.604797 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.604798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.605326 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.605327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.605976 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.605977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.606067 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.606566 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.606571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.607985 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.610077 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.610078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.610079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.610324 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.610543 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.610544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.610545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.610620 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.610905 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.610906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.610964 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.611037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.611115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:23:13.611351 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.603750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.4.64.124:43281)
0810 15:23:13.603814 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.603816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.603827 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:23:13.603848 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.603849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.603861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.603935 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.604173 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.604704 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.605484 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.605860 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.605958 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.608094 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.608095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.608096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.608108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.609985 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.610410 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.610410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.610411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.610468 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.610724 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.610724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.610725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.610833 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.610889 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.611093 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.611178 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:23:13.612380 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.603009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.4.64.126:44205)
0810 15:23:13.603402 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.603404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.603412 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:23:13.603437 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.603438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.603448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.603530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.604155 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.604627 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.605327 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.605625 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.605734 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.608895 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.608896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.608896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.608903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.610891 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.611343 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.611344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.611345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.611411 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.611634 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.611635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.611636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.611804 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.612000 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.612201 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.612291 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367}
I20260810 15:23:13.613334 20052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327: Bootstrap starting.
I20260810 15:23:13.614388 20052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:13.614739 20052 log.cc:824] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:13.615631 20052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327: No bootstrap required, opened a new log
I20260810 15:23:13.617835 20052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.618014 20052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:13.618095 20052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ea8964523c47749cfb5392255ab327, State: Initialized, Role: FOLLOWER
I20260810 15:23:13.618294 20052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.618638 20052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:13.618621 20059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.618741 20059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
W20260810 15:23:13.620810 20070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:23:13.620922 20070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:23:13.623698 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.615066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.4.64.125:43573)
0810 15:23:13.615136 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.615138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.615156 (+ 18us) server_negotiation.cc:407] Connection header received
0810 15:23:13.615179 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.615180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.615188 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.615243 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.615825 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.616603 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.617279 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.617607 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.617724 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.620217 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.620219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.620219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.620227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.622121 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.622589 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.622590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.622590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.622674 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.622918 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.622919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.622919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.623100 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.623280 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.623521 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.623604 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:23:13.621575 19965 master_runner.cc:186] Error getting master registration for 127.4.64.126:44205: OK, instance_id { permanent_uuid: "f936889662694926b0ecf941b9eb542e" instance_seqno: 1786375393211805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:23:13.624864 19965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:23:13.625241 19958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } attempt: 3
I20260810 15:23:13.626474 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:23:13.633334 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.625503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46280)
0810 15:23:13.625564 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.625575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.625639 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.625894 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.625895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.626057 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.626094 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.626096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.626933 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.626934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.627462 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.627464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.627973 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.627974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.628034 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.628468 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.628474 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.630119 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.632028 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.632029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.632029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.632252 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.632503 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.632504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.632504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.632570 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.632931 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.632932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.632991 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.633074 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.633185 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 15:23:13.633939 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.625712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.4.64.124:43281)
0810 15:23:13.625752 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.625754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.625761 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:23:13.625779 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.625780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.625789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.625839 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.626185 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.626849 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.627577 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.627876 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.627963 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.630208 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.630209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.630209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.630217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.631941 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.632359 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.632360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.632361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.632421 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.632646 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.632647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.632648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.632829 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.632900 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.633749 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.633837 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:23:13.635742 19958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: Bootstrap starting.
I20260810 15:23:13.636420 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.628335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.4.64.126:44205)
0810 15:23:13.628404 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.628407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.628573 (+ 166us) server_negotiation.cc:407] Connection header received
0810 15:23:13.628682 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.628683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.628694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.628725 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.629535 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.630246 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.631654 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.631994 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.632037 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.633589 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.633590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.633591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.633597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.635263 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.635632 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.635633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.635633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.635696 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.635903 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.635905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.635905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.636025 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.636128 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.636237 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.636319 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:23:13.636555 19958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:13.636911 19958 log.cc:824] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:13.637900 19958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: No bootstrap required, opened a new log
I20260810 15:23:13.639426 19958 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.639577 19958 raft_consensus.cc:374] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:13.639658 19958 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97061cafc584dd5a1c6d2b338b9bf17, State: Initialized, Role: FOLLOWER
I20260810 15:23:13.639799 19958 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.640209 19958 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 15:23:13.642199 20093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:23:13.642314 20093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:23:13.645485 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.628478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.4.64.125:43573)
0810 15:23:13.628517 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.628519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.629229 (+ 710us) server_negotiation.cc:407] Connection header received
0810 15:23:13.629336 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.629337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.629348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.629379 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.632334 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.633154 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.639420 (+ 6266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.639798 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.639845 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.641978 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.641979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.641980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.641988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.643778 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.644513 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.644514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.644514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.644575 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.644801 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.644802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.644803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.644942 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.645027 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.645278 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.645390 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:23:13.642527 19874 master_runner.cc:186] Error getting master registration for 127.4.64.126:44205: OK, instance_id { permanent_uuid: "f936889662694926b0ecf941b9eb542e" instance_seqno: 1786375393211805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:23:13.645936 19874 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:23:13.647172 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.628845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.4.64.124:43281)
0810 15:23:13.629112 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.629114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.630374 (+ 1260us) server_negotiation.cc:407] Connection header received
0810 15:23:13.630397 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.630398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.630409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.630471 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.631250 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.632205 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.638875 (+ 6670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.639247 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.639328 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.643825 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.643827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.643827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.643835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.645774 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.646193 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.646194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.646195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.646284 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.646537 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.646538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.646539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.646686 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.646799 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.646943 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.647032 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":78,"threads_started":1}
I20260810 15:23:13.649744 20081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.649863 20081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:23:13.659407 19867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } attempt: 3
Aug 10 15:23:13 dist-test-slave-9zdj krb5kdc[19766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:23:13.668372 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.659717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:43281 (local address 127.0.0.1:46304)
0810 15:23:13.659774 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:13.659785 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:13.659848 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:13.660050 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:13.660051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:13.660249 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:13.660304 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.660306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.661281 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.661282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.662208 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:13.662211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.662741 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.662742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:13.662813 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.663347 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:13.663353 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:13.664993 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:13.667157 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.667159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.667159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.667384 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.667590 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:13.667590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:13.667591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:13.667646 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:13.667997 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:13.667998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:13.668051 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 15:23:13.668125 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 15:23:13.668232 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:23:13.668406 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:13.659786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.4.64.124:43281)
0810 15:23:13.659834 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:13.659836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:13.659846 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:23:13.659939 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:13.659941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:13.659954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:13.660001 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:13.660390 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.661185 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.662307 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:13.662651 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:13.662727 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:13.665089 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:13.665091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:13.665091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:13.665099 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:13.667056 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.667466 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.667467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.667467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.667524 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:13.667707 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:13.667707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:13.667708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:13.667894 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:13.667967 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:13.668190 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 15:23:13.668298 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:23:13.670450 19867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: Bootstrap starting.
I20260810 15:23:13.671324 19867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:13.671669 19867 log.cc:824] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:13.672703 19867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: No bootstrap required, opened a new log
I20260810 15:23:13.676204 19867 raft_consensus.cc:348] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.676453 19867 raft_consensus.cc:374] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:13.676548 19867 raft_consensus.cc:729] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f936889662694926b0ecf941b9eb542e, State: Initialized, Role: FOLLOWER
I20260810 15:23:13.676775 19867 consensus_queue.cc:260] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.677222 19867 sys_catalog.cc:564] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:13.677224 20097 sys_catalog.cc:455] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.677443 20097 sys_catalog.cc:458] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:23:13.679512 20108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:23:13.679575 20108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:23:13.682940 19784 master_runner.cc:263] No leader master found from master f936889662694926b0ecf941b9eb542e
I20260810 15:23:13.683007 19784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 15:23:13.712677 19810 tablet.cc:2369] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:13.772054 20059 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:13.772209 20059 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.772500 20059 leader_election.cc:288] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f936889662694926b0ecf941b9eb542e (127.4.64.126:44205), d97061cafc584dd5a1c6d2b338b9bf17 (127.4.64.125:43573)
I20260810 15:23:13.772874 19836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7ea8964523c47749cfb5392255ab327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f936889662694926b0ecf941b9eb542e" is_pre_election: true
I20260810 15:23:13.773049 19836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7ea8964523c47749cfb5392255ab327 in term 0.
I20260810 15:23:13.773509 19989 leader_election.cc:302] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7ea8964523c47749cfb5392255ab327, f936889662694926b0ecf941b9eb542e; no voters:
I20260810 15:23:13.773797 20059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:23:13.773866 20059 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:23:13.773892 20059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:13.774972 20059 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.775116 20059 leader_election.cc:288] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [CANDIDATE]: Term 1 election: Requested vote from peers f936889662694926b0ecf941b9eb542e (127.4.64.126:44205), d97061cafc584dd5a1c6d2b338b9bf17 (127.4.64.125:43573)
I20260810 15:23:13.775458 19836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7ea8964523c47749cfb5392255ab327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f936889662694926b0ecf941b9eb542e"
I20260810 15:23:13.775539 19836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:13.776480 19836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7ea8964523c47749cfb5392255ab327 in term 1.
I20260810 15:23:13.776753 19989 leader_election.cc:302] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7ea8964523c47749cfb5392255ab327, f936889662694926b0ecf941b9eb542e; no voters:
I20260810 15:23:13.777172 19926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7ea8964523c47749cfb5392255ab327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" is_pre_election: true
I20260810 15:23:13.777369 19926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7ea8964523c47749cfb5392255ab327 in term 0.
I20260810 15:23:13.777536 19925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7ea8964523c47749cfb5392255ab327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97061cafc584dd5a1c6d2b338b9bf17"
I20260810 15:23:13.777599 19925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:13.779175 19925 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7ea8964523c47749cfb5392255ab327 in term 1.
I20260810 15:23:13.779743 20059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:13.779928 20059 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [term 1 LEADER]: Becoming Leader. State: Replica: b7ea8964523c47749cfb5392255ab327, State: Running, Role: LEADER
I20260810 15:23:13.780019 20059 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } }
I20260810 15:23:13.780408 20059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ea8964523c47749cfb5392255ab327. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.780486 20059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:13.780859 20112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:13.781117 20112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:13.783519 19925 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [term 1 FOLLOWER]: Refusing update from remote peer b7ea8964523c47749cfb5392255ab327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:13.784008 20059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:13.786615 20081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ea8964523c47749cfb5392255ab327. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.786733 20081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:23:13.787457 19836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [term 1 FOLLOWER]: Refusing update from remote peer b7ea8964523c47749cfb5392255ab327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:13.787798 20059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:13.789134 20097 sys_catalog.cc:455] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ea8964523c47749cfb5392255ab327. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.789234 20097 sys_catalog.cc:458] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:23:13.790714 20110 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.790807 20110 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:13.791093 20059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.791160 20059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:13.791661 20081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.791747 20081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:23:13.792276 20097 sys_catalog.cc:455] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7ea8964523c47749cfb5392255ab327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f936889662694926b0ecf941b9eb542e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44205 } } peers { permanent_uuid: "d97061cafc584dd5a1c6d2b338b9bf17" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43573 } } peers { permanent_uuid: "b7ea8964523c47749cfb5392255ab327" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 43281 } } }
I20260810 15:23:13.792361 20097 sys_catalog.cc:458] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:23:13.792749 20113 mvcc.cc:205] Tried to move back new op lower bound from 7316993612934426624 to 7316993612923531264. Current Snapshot: MvccSnapshot[applied={T|T < 7316993612934426624}]
I20260810 15:23:13.792886 20117 mvcc.cc:205] Tried to move back new op lower bound from 7316993612934426624 to 7316993612923531264. Current Snapshot: MvccSnapshot[applied={T|T < 7316993612934426624}]
I20260810 15:23:13.793084 20114 mvcc.cc:205] Tried to move back new op lower bound from 7316993612934426624 to 7316993612923531264. Current Snapshot: MvccSnapshot[applied={T|T < 7316993612934426624}]
I20260810 15:23:13.794657 20112 catalog_manager.cc:1347] Generated new cluster ID: 0159ec2672994e58bde3035230322d27
I20260810 15:23:13.794705 20112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:23:13.817713 20112 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:23:13.818676 20112 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:13.830385 20112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327: Generated new TSK 0
I20260810 15:23:13.830838 20112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:23:14.642920 20093 catalog_manager.cc:1259] Loaded cluster ID: 0159ec2672994e58bde3035230322d27
I20260810 15:23:14.643004 20093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: loading cluster ID for follower catalog manager: success
I20260810 15:23:14.643810 20093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: acquiring CA information for follower catalog manager: success
I20260810 15:23:14.644073 20093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d97061cafc584dd5a1c6d2b338b9bf17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:23:14.680106 20108 catalog_manager.cc:1259] Loaded cluster ID: 0159ec2672994e58bde3035230322d27
I20260810 15:23:14.680186 20108 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: loading cluster ID for follower catalog manager: success
I20260810 15:23:14.681002 20108 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: acquiring CA information for follower catalog manager: success
I20260810 15:23:14.681277 20108 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f936889662694926b0ecf941b9eb542e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:23:14.895784 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19784
I20260810 15:23:14.906311 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19874
I20260810 15:23:14.909981 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:14.909694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44205 (local address 127.0.0.1:60046)
0810 15:23:14.909854 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:14.909919 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
W20260810 15:23:14.910210 19989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44205: connect: Connection refused (error 111)
I20260810 15:23:14.910732 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:14.910590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44205 (local address 127.0.0.1:60050)
0810 15:23:14.910652 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:14.910684 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 15:23:14.910851 19989 consensus_peers.cc:487] T 00000000000000000000000000000000 P b7ea8964523c47749cfb5392255ab327 -> Peer f936889662694926b0ecf941b9eb542e (127.4.64.126:44205): Couldn't send request to peer f936889662694926b0ecf941b9eb542e. Status: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 15:23:14.914980 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 19965
2026-08-10T15:23:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3110 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:23:14 dist-test-slave-9zdj krb5kdc[20133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:23:14 dist-test-slave-9zdj krb5kdc[20133](info): set up 2 sockets
Aug 10 15:23:14 dist-test-slave-9zdj krb5kdc[20133](info): commencing operation
krb5kdc: starting...
W20260810 15:23:16.056943 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:23:16 dist-test-slave-9zdj krb5kdc[20133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:23:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:23:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:23:16.197866 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44407
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:16.283576 20150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:16.283849 20150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:16.283891 20150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:16.285224 20150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:16.285307 20150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:16.285331 20150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:16.285351 20150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:16.285386 20150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:16.286959 20150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:16.287261 20150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:16.287515 20150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:16.292868 20159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:16.292917 20156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:16.292946 20150 server_base.cc:1034] running on GCE node
W20260810 15:23:16.292898 20155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:16.293505 20150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:16.293859 20150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:16.295035 20150 hybrid_clock.cc:648] HybridClock initialized: now 1786375396295015 us; error 39 us; skew 500 ppm
Aug 10 15:23:16 dist-test-slave-9zdj krb5kdc[20133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:16.297783 20150 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:16.298251 20150 webserver.cc:460] Webserver started at http://127.4.64.126:45079/ using document root <none> and password file <none>
I20260810 15:23:16.298529 20150 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:16.298602 20150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:16.298764 20150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:16.299871 20150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f"
format_stamp: "Formatted at 2026-08-10 15:23:16 on dist-test-slave-9zdj"
server_key: "1de23b1624b6456e39d8f262a55a1d17"
server_key_iv: "098b5c608ec1192b08b33383a161a85e"
server_key_version: "encryptionkey@0"
I20260810 15:23:16.300269 20150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f"
format_stamp: "Formatted at 2026-08-10 15:23:16 on dist-test-slave-9zdj"
server_key: "1de23b1624b6456e39d8f262a55a1d17"
server_key_iv: "098b5c608ec1192b08b33383a161a85e"
server_key_version: "encryptionkey@0"
I20260810 15:23:16.301748 20150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:23:16.302623 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:16.302819 20150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:16.302934 20150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f"
format_stamp: "Formatted at 2026-08-10 15:23:16 on dist-test-slave-9zdj"
server_key: "1de23b1624b6456e39d8f262a55a1d17"
server_key_iv: "098b5c608ec1192b08b33383a161a85e"
server_key_version: "encryptionkey@0"
I20260810 15:23:16.303016 20150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:16.318611 20150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:16.320664 20150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:16.320852 20150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:16.325232 20150 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44407
I20260810 15:23:16.325266 20229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44407 every 8 connection(s)
I20260810 15:23:16.325673 20150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:16.326646 20230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:16.328877 20230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f: Bootstrap starting.
I20260810 15:23:16.329424 20230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:16.329777 20230 log.cc:824] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:16.330502 20230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f: No bootstrap required, opened a new log
I20260810 15:23:16.332051 20230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44407 } }
I20260810 15:23:16.332194 20230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:16.332217 20230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df3d9cf3a4c4e598fd0a5bf89bbd95f, State: Initialized, Role: FOLLOWER
I20260810 15:23:16.332376 20230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44407 } }
I20260810 15:23:16.332463 20230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:16.332487 20230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:16.332516 20230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:16.333004 20230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44407 } }
I20260810 15:23:16.333089 20230 leader_election.cc:302] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df3d9cf3a4c4e598fd0a5bf89bbd95f; no voters:
I20260810 15:23:16.333216 20230 leader_election.cc:288] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:23:16.333318 20233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:16.333526 20233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [term 1 LEADER]: Becoming Leader. State: Replica: 0df3d9cf3a4c4e598fd0a5bf89bbd95f, State: Running, Role: LEADER
I20260810 15:23:16.333532 20230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:16.333673 20233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44407 } }
I20260810 15:23:16.334278 20235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df3d9cf3a4c4e598fd0a5bf89bbd95f. Latest consensus state: current_term: 1 leader_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44407 } } }
I20260810 15:23:16.334373 20235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:16.334542 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 20150
I20260810 15:23:16.334619 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:23:16.334771 20245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:16.334843 4353 external_mini_cluster.cc:1428] Setting key 37c8113c0e9c6f4413f2d8488f70373d
I20260810 15:23:16.334774 20234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df3d9cf3a4c4e598fd0a5bf89bbd95f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44407 } } }
I20260810 15:23:16.335315 20234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:16.335688 20245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:16.337714 20245 catalog_manager.cc:1347] Generated new cluster ID: 867e4a08fae941a28a83a3e677d3f1fd
I20260810 15:23:16.337771 20245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:23:16 dist-test-slave-9zdj krb5kdc[20133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:16.347242 20258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:16.335315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.4.64.126:44407)
0810 15:23:16.335481 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:16.335490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:16.335518 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:23:16.335712 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:16.335720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:16.335776 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:16.335932 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:16.336712 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:16.338150 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:16.339272 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:16.339911 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:16.340035 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:16.342995 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:16.343002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:16.343002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:16.343019 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:16.345161 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:16.345588 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:16.345592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:16.345593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:16.345764 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:16.345991 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:16.345992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:16.345992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:16.346361 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:16.346639 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:16.346824 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 15:23:16.346983 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260810 15:23:16.347252 20245 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:23:16.347940 20245 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:16.348335 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:23:16.354056 20245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df3d9cf3a4c4e598fd0a5bf89bbd95f: Generated new TSK 0
I20260810 15:23:16.354250 20245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:23:16.403090 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 20150
2026-08-10T15:23:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1486 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:23:16 dist-test-slave-9zdj krb5kdc[20263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:23:16 dist-test-slave-9zdj krb5kdc[20263](info): set up 2 sockets
Aug 10 15:23:16 dist-test-slave-9zdj krb5kdc[20263](info): commencing operation
krb5kdc: starting...
W20260810 15:23:17.941506 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.517s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:23:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:23:18.162447 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44711
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:18.331348 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:18.332012 20280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:18.332093 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:18.334273 20280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:18.334401 20280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:18.334467 20280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:18.334546 20280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:18.334589 20280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:18.337218 20280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:18.337762 20280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:18.338142 20280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:18.346956 20287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.346959 20286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.347015 20290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:18.347484 20280 server_base.cc:1034] running on GCE node
I20260810 15:23:18.347918 20280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:18.348387 20280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:18.349689 20280 hybrid_clock.cc:648] HybridClock initialized: now 1786375398349554 us; error 87 us; skew 500 ppm
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:18.354409 20280 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:18.355337 20280 webserver.cc:460] Webserver started at http://127.4.64.126:37035/ using document root <none> and password file <none>
I20260810 15:23:18.355895 20280 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:18.356084 20280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:18.356534 20280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:18.358479 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e94c519bfc7c42fe9d30a821b3b0b056"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "173bbd69bfa43056aeb13a5765be6fd5"
server_key_iv: "e15bf77151fbb9d5359941abca3099b8"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.359107 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e94c519bfc7c42fe9d30a821b3b0b056"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "173bbd69bfa43056aeb13a5765be6fd5"
server_key_iv: "e15bf77151fbb9d5359941abca3099b8"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.361828 20280 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 15:23:18.363369 20300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.363698 20280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:18.363884 20280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e94c519bfc7c42fe9d30a821b3b0b056"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "173bbd69bfa43056aeb13a5765be6fd5"
server_key_iv: "e15bf77151fbb9d5359941abca3099b8"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.364012 20280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:18.374666 20280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:18.378293 20280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:18.378518 20280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:18.386148 20280 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44711
I20260810 15:23:18.386173 20362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44711 every 8 connection(s)
I20260810 15:23:18.386724 20280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:18.388480 20363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:18.392159 20363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap starting.
I20260810 15:23:18.393036 20363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:18.393363 20363 log.cc:824] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:18.394500 20363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: No bootstrap required, opened a new log
I20260810 15:23:18.396584 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 20280
I20260810 15:23:18.396778 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:23:18.397189 4353 external_mini_cluster.cc:1428] Setting key 3d119743958e1a7c849b107d4f9445ff
I20260810 15:23:18.397866 20363 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:18.398204 20363 raft_consensus.cc:374] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:18.398276 20363 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94c519bfc7c42fe9d30a821b3b0b056, State: Initialized, Role: FOLLOWER
I20260810 15:23:18.398564 20363 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:18.398747 20363 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:18.398797 20363 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:18.398900 20363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:18.400806 20363 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:18.400961 20363 leader_election.cc:302] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94c519bfc7c42fe9d30a821b3b0b056; no voters:
I20260810 15:23:18.401182 20363 leader_election.cc:288] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:23:18.401262 20368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:18.401563 20368 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 LEADER]: Becoming Leader. State: Replica: e94c519bfc7c42fe9d30a821b3b0b056, State: Running, Role: LEADER
I20260810 15:23:18.401705 20363 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:18.401808 20368 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:18.402402 20368 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94c519bfc7c42fe9d30a821b3b0b056. Latest consensus state: current_term: 1 leader_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } } }
I20260810 15:23:18.402570 20368 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:18.402418 20370 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } } }
I20260810 15:23:18.404286 20370 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: This master's current role is: LEADER
W20260810 15:23:18.404811 20385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:23:18.404914 20385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:23:18.405057 20377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:18.405403 20377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:18.408231 20377 catalog_manager.cc:1347] Generated new cluster ID: d6660c12d8ea4949a28c7c6e3c152dcb
I20260810 15:23:18.408313 20377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:18.416572 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:18.398206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.4.64.126:44711)
0810 15:23:18.398553 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:18.398563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:18.398627 (+ 64us) server_negotiation.cc:407] Connection header received
0810 15:23:18.398833 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:18.398853 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:18.398963 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:18.399273 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:18.400226 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.402481 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.403855 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.404988 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.405211 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:18.410083 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:18.410096 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:18.410096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:18.410115 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:18.413575 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:18.414396 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:18.414400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:18.414401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:18.414607 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:18.414932 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:18.414933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:18.414934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:18.415440 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:18.415723 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:18.415968 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 15:23:18.416206 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":148,"threads_started":1}
I20260810 15:23:18.422015 20377 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:23:18.422812 20377 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:18.429195 20377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Generated new TSK 0
I20260810 15:23:18.429533 20377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:23:18.469686 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:44711
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:18.567312 20390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:18.567528 20390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:18.567553 20390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:18.568739 20390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:18.568789 20390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:18.568816 20390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:18.570489 20390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:18.570739 20390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:18.571000 20390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:18.577591 20397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.577581 20395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.577569 20400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:18.578374 20390 server_base.cc:1034] running on GCE node
I20260810 15:23:18.578619 20390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:18.578898 20390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:18.580076 20390 hybrid_clock.cc:648] HybridClock initialized: now 1786375398580061 us; error 32 us; skew 500 ppm
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:18.583024 20390 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:18.583509 20390 webserver.cc:460] Webserver started at http://127.4.64.65:39131/ using document root <none> and password file <none>
I20260810 15:23:18.583811 20390 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:18.583886 20390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:18.584049 20390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:18.585198 20390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "17d3da8ab7445a8044b8f82e0cd24dca"
server_key_iv: "2a1966fa1f715d94f4d51006b3a98f3b"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.585673 20390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "17d3da8ab7445a8044b8f82e0cd24dca"
server_key_iv: "2a1966fa1f715d94f4d51006b3a98f3b"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.587230 20390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:18.588075 20408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.588263 20390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:18.588375 20390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "17d3da8ab7445a8044b8f82e0cd24dca"
server_key_iv: "2a1966fa1f715d94f4d51006b3a98f3b"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.588466 20390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:18.604132 20390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:18.606233 20390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:18.606364 20390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:18.606556 20390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:18.606910 20390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:18.606947 20390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.606995 20390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:18.607020 20390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.613330 20390 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:34585
I20260810 15:23:18.613420 20541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:34585 every 8 connection(s)
I20260810 15:23:18.613813 20390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:18.618163 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 20390
I20260810 15:23:18.618292 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:23:18.618507 4353 external_mini_cluster.cc:1428] Setting key 3df9f0a09d6e70aa6e92d20426f867e0
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:18.625154 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:18.614751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:49607 (local address 127.4.64.126:44711)
0810 15:23:18.614827 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:18.614830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:18.615144 (+ 314us) server_negotiation.cc:407] Connection header received
0810 15:23:18.615625 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:18.615626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:18.615642 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:18.615695 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:18.616791 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.617312 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.618774 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.619052 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.619131 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:18.621861 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:18.621863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:18.621863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:18.621874 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:18.623568 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:18.623978 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:18.623979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:18.623979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:18.624047 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:18.624297 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:18.624298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:18.624298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:18.624502 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:18.624585 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:18.624924 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 15:23:18.625020 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:23:18.625187 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:18.614720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:49607)
0810 15:23:18.614992 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:18.615035 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:18.615538 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:18.615769 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:18.615773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:18.616479 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:18.616696 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:18.616700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.617400 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.617401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:18.618669 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:18.618673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.619166 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.619167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:18.619400 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:18.619965 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:18.619976 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:18.621758 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:18.623658 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:18.623665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:18.623667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:18.623897 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:18.624119 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:18.624119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:18.624120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:18.624224 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:18.624578 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:18.624583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:18.624772 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 15:23:18.624867 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 15:23:18.625000 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":135,"threads_started":1}
I20260810 15:23:18.625792 20542 heartbeater.cc:347] Connected to a master server at 127.4.64.126:44711
I20260810 15:23:18.625917 20542 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:18.626144 20542 heartbeater.cc:511] Master 127.4.64.126:44711 requested a full tablet report, sending...
I20260810 15:23:18.626749 20321 ts_manager.cc:194] Registered new tserver with Master: e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585)
I20260810 15:23:18.627388 20321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:49607
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:23:18.653364 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:44711
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:18.753607 20549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:18.753947 20549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:18.753991 20549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:18.755270 20549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:18.755311 20549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:18.755339 20549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:18.756837 20549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:18.757051 20549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:18.757267 20549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:18.762673 20557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.762689 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:18.762841 20549 server_base.cc:1034] running on GCE node
W20260810 15:23:18.762722 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:18.763082 20549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:18.763319 20549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:18.764473 20549 hybrid_clock.cc:648] HybridClock initialized: now 1786375398764459 us; error 29 us; skew 500 ppm
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:18.767207 20549 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:18.767758 20549 webserver.cc:460] Webserver started at http://127.4.64.66:43645/ using document root <none> and password file <none>
I20260810 15:23:18.767999 20549 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:18.768046 20549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:18.768196 20549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:18.769347 20549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "410d8070b5c8497da243188b8add0f6b"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "a2355339f56ae99d66176df25995ff1e"
server_key_iv: "b679fb5c9b772af332a4ccd9c48ac24a"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.769762 20549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "410d8070b5c8497da243188b8add0f6b"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "a2355339f56ae99d66176df25995ff1e"
server_key_iv: "b679fb5c9b772af332a4ccd9c48ac24a"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.771230 20549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:18.772146 20569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.772398 20549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:18.772541 20549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "410d8070b5c8497da243188b8add0f6b"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "a2355339f56ae99d66176df25995ff1e"
server_key_iv: "b679fb5c9b772af332a4ccd9c48ac24a"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.772641 20549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:18.809988 20549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:18.812844 20549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:18.813083 20549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:18.813442 20549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:18.813895 20549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:18.813962 20549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.814002 20549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:18.814033 20549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.821108 20549 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38903
I20260810 15:23:18.821165 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38903 every 8 connection(s)
I20260810 15:23:18.821578 20549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:18.830713 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 20549
I20260810 15:23:18.830821 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:23:18.831043 4353 external_mini_cluster.cc:1428] Setting key 881f7913df40c3b74c3d47d873bfd534
I20260810 15:23:18.833120 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:18.822476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:49809 (local address 127.4.64.126:44711)
0810 15:23:18.822544 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:18.822546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:18.822788 (+ 242us) server_negotiation.cc:407] Connection header received
0810 15:23:18.823319 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:18.823320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:18.823331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:18.823396 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:18.824577 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.825144 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.826337 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.826533 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.826593 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:18.829377 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:18.829379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:18.829379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:18.829387 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:18.831316 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:18.831848 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:18.831849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:18.831850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:18.831963 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:18.832345 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:18.832345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:18.832346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:18.832481 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:18.832544 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:18.832958 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 15:23:18.833047 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:23:18.833431 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:18.822449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:49809)
0810 15:23:18.822621 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:18.822670 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:18.823226 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:18.823457 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:18.823461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:18.824228 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:18.824462 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:18.824467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.825260 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.825261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:18.826247 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:18.826250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:18.826625 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:18.826626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:18.826793 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:18.827496 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:18.827507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:18.829236 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:18.831452 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:18.831460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:18.831463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:18.831731 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:18.832059 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:18.832060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:18.832060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:18.832257 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:18.832557 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:18.832565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:18.832769 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 15:23:18.832900 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 15:23:18.833113 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260810 15:23:18.834030 20708 heartbeater.cc:347] Connected to a master server at 127.4.64.126:44711
I20260810 15:23:18.834204 20708 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:18.834466 20708 heartbeater.cc:511] Master 127.4.64.126:44711 requested a full tablet report, sending...
I20260810 15:23:18.834971 20321 ts_manager.cc:194] Registered new tserver with Master: 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
I20260810 15:23:18.835561 20321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:49809
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:23:18.865674 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:44711
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:18.964501 20713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:18.964762 20713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:18.964792 20713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:18.966223 20713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:18.966274 20713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:18.966305 20713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:18.967962 20713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:18.968304 20713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:18.968590 20713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:18.974730 20720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.974730 20719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:18.974778 20722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:18.975766 20713 server_base.cc:1034] running on GCE node
I20260810 15:23:18.976053 20713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:18.976404 20713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:18.977594 20713 hybrid_clock.cc:648] HybridClock initialized: now 1786375398977559 us; error 53 us; skew 500 ppm
Aug 10 15:23:18 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:18.980837 20713 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:18.981467 20713 webserver.cc:460] Webserver started at http://127.4.64.67:36785/ using document root <none> and password file <none>
I20260810 15:23:18.981787 20713 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:18.981870 20713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:18.982053 20713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:18.983366 20713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "e555807f830a490a9a571637e90b9f57"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "cfb046667d05aa7a015828bf37dd07b3"
server_key_iv: "30341a270a4dc6456993b4788deca67a"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.983808 20713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "e555807f830a490a9a571637e90b9f57"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "cfb046667d05aa7a015828bf37dd07b3"
server_key_iv: "30341a270a4dc6456993b4788deca67a"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.985544 20713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:23:18.986603 20729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:18.986897 20713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:18.987038 20713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "e555807f830a490a9a571637e90b9f57"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "cfb046667d05aa7a015828bf37dd07b3"
server_key_iv: "30341a270a4dc6456993b4788deca67a"
server_key_version: "encryptionkey@0"
I20260810 15:23:18.987126 20713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:19.000826 20713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:19.003921 20713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:19.004175 20713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:19.004503 20713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:19.004909 20713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:19.004971 20713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.005017 20713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:19.005052 20713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.011948 20713 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:44285
I20260810 15:23:19.011999 20860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:44285 every 8 connection(s)
I20260810 15:23:19.012398 20713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:23:19.013301 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 20713
I20260810 15:23:19.013424 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:23:19.013675 4353 external_mini_cluster.cc:1428] Setting key e59a6c4c572f80502b7202951df72d99
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:19.023883 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.013361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:53427 (local address 127.4.64.126:44711)
0810 15:23:19.013596 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.013598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.013680 (+ 82us) server_negotiation.cc:407] Connection header received
0810 15:23:19.014495 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.014496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.014511 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.014601 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.015832 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.016384 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.017536 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.017791 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.017854 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.020490 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.020492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.020493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.020500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.022274 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.022766 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.022767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.022767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.022848 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.023190 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.023191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.023191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.023332 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.023394 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.023716 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.023809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260810 15:23:19.024087 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.013274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:53427)
0810 15:23:19.013502 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.013564 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.014366 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.014684 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.014688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.015454 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:19.015690 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.015695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.016491 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.016491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.017443 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.017445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.017872 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.017873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.017988 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.018588 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:19.018598 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:19.020376 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:19.022372 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.022380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.022382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.022651 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.022971 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.022973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.022973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.023120 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.023401 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:19.023407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:19.023547 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.023648 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.023800 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260810 15:23:19.024773 20862 heartbeater.cc:347] Connected to a master server at 127.4.64.126:44711
I20260810 15:23:19.024925 20862 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:19.025202 20862 heartbeater.cc:511] Master 127.4.64.126:44711 requested a full tablet report, sending...
I20260810 15:23:19.025672 20321 ts_manager.cc:194] Registered new tserver with Master: e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285)
I20260810 15:23:19.025941 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:23:19.026249 20321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:53427
I20260810 15:23:19.034276 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.027481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.4.64.126:44711)
0810 15:23:19.027526 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.027529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.027619 (+ 90us) server_negotiation.cc:407] Connection header received
0810 15:23:19.027671 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.027672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.027681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.027736 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.028462 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.028952 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.029865 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.030077 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.030156 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.031499 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.031501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.031501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.031507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.032908 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.033361 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.033362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.033362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.033423 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.033760 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.033761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.033762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.033923 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.034015 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.034138 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.034206 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:23:19.036530 20321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:23:19.038415 20321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:23:19.046460 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.041760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.4.64.65:34585)
0810 15:23:19.041958 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.041964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.041976 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:23:19.042025 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.042027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.042046 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.042139 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.042467 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.043473 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.045165 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.045936 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.046023 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.046091 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.046298 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.046347 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260810 15:23:19.046589 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.041478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38903 (local address 127.0.0.1:37956)
0810 15:23:19.042399 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.042412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.042469 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.042749 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.042750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.042753 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.042800 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.042802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.044238 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.044238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.045166 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.045169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.046266 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.046267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.046345 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.046350 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.046403 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.046475 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":49,"thread_start_us":109,"threads_started":1}
I20260810 15:23:19.046883 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.041819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.4.64.66:38903)
0810 15:23:19.042038 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.042044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.042566 (+ 522us) server_negotiation.cc:407] Connection header received
0810 15:23:19.042619 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.042622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.042640 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.042733 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.042905 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.044108 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.045286 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.046070 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.046149 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.046227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.046730 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.046783 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260810 15:23:19.047056 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.041432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:34585 (local address 127.0.0.1:37998)
0810 15:23:19.041571 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.041644 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.041733 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.042263 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.042267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.042270 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.042360 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.042363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.043603 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.043604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.045036 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.045040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.046041 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.046043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.046160 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.046179 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.046811 (+ 632us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.046861 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260810 15:23:19.047223 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.041478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:44285 (local address 127.0.0.1:39860)
0810 15:23:19.041649 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.041659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.041734 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.042571 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.042572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.042574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.042672 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.042674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.044205 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.044207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.045583 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.045586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.046960 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.046962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.047037 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.047043 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.047093 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.047140 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":66,"thread_start_us":102,"threads_started":1}
I20260810 15:23:19.047458 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.042053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.4.64.67:44285)
0810 15:23:19.042369 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.042374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.042383 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:23:19.042440 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.042443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.042460 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.042557 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.042772 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.044055 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.045723 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.046515 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.046957 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.047215 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.047247 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.047297 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":223,"threads_started":1}
I20260810 15:23:19.047866 20625 tablet_service.cc:1467] Processing CreateTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 (DEFAULT_TABLE table=test-table [id=19abbf6f38e4497e9c7f73227e03009d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:19.048137 20781 tablet_service.cc:1467] Processing CreateTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 (DEFAULT_TABLE table=test-table [id=19abbf6f38e4497e9c7f73227e03009d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:19.048452 20625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1dc4b1cb114146af5c8a8e4c329737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:19.048527 20781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1dc4b1cb114146af5c8a8e4c329737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:19.052744 20882 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Bootstrap starting.
I20260810 15:23:19.053051 20883 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Bootstrap starting.
I20260810 15:23:19.053691 20464 tablet_service.cc:1467] Processing CreateTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 (DEFAULT_TABLE table=test-table [id=19abbf6f38e4497e9c7f73227e03009d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:19.053895 20883 tablet_bootstrap.cc:654] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:19.053954 20882 tablet_bootstrap.cc:654] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:19.054103 20464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1dc4b1cb114146af5c8a8e4c329737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:19.054821 20883 log.cc:824] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.054884 20882 log.cc:824] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.055811 20882 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: No bootstrap required, opened a new log
I20260810 15:23:19.055891 20882 ts_tablet_manager.cc:1397] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:23:19.055989 20883 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: No bootstrap required, opened a new log
I20260810 15:23:19.056130 20883 ts_tablet_manager.cc:1397] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:23:19.058336 20882 raft_consensus.cc:348] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.058476 20886 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap starting.
I20260810 15:23:19.058563 20882 raft_consensus.cc:374] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:19.058638 20882 raft_consensus.cc:729] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410d8070b5c8497da243188b8add0f6b, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.058794 20882 consensus_queue.cc:260] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.059131 20882 ts_tablet_manager.cc:1428] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:23:19.059403 20886 tablet_bootstrap.cc:654] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:19.059561 20883 raft_consensus.cc:348] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.059734 20883 raft_consensus.cc:374] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:19.059760 20883 raft_consensus.cc:729] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e555807f830a490a9a571637e90b9f57, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.059906 20883 consensus_queue.cc:260] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.060063 20886 log.cc:824] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.060266 20862 heartbeater.cc:503] Master 127.4.64.126:44711 was elected leader, sending a full tablet report...
I20260810 15:23:19.060597 20708 heartbeater.cc:503] Master 127.4.64.126:44711 was elected leader, sending a full tablet report...
I20260810 15:23:19.060882 20883 ts_tablet_manager.cc:1428] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 15:23:19.061244 20886 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: No bootstrap required, opened a new log
I20260810 15:23:19.061322 20886 ts_tablet_manager.cc:1397] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:23:19.063064 20886 raft_consensus.cc:348] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.063231 20886 raft_consensus.cc:374] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:19.063257 20886 raft_consensus.cc:729] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822f28d3a7741a3b8eb56e3aee2c460, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.063388 20886 consensus_queue.cc:260] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.063664 20886 ts_tablet_manager.cc:1428] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 15:23:19.063860 20542 heartbeater.cc:503] Master 127.4.64.126:44711 was elected leader, sending a full tablet report...
W20260810 15:23:19.072233 20709 tablet.cc:2369] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:23:19.114976 20543 tablet.cc:2369] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:19.210630 20887 raft_consensus.cc:482] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:19.210789 20887 raft_consensus.cc:504] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.211182 20887 leader_election.cc:288] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585), e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285)
I20260810 15:23:19.215508 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.211533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:45689 (local address 127.4.64.65:34585)
0810 15:23:19.211586 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.211588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.211762 (+ 174us) server_negotiation.cc:407] Connection header received
0810 15:23:19.211795 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.211796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.211811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.211877 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.212038 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.212928 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.214523 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.215087 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.215177 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.215271 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.215374 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.215440 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:23:19.215523 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.211566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:50805 (local address 127.4.64.67:44285)
0810 15:23:19.211613 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.211615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.211663 (+ 48us) server_negotiation.cc:407] Connection header received
0810 15:23:19.211693 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.211693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.211703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.211816 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.212035 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.212885 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.214523 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.215107 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.215177 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.215191 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.215424 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.215464 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:23:19.215505 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.211435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:44285 (local address 127.4.64.66:50805)
0810 15:23:19.211497 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.211510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.211558 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.211836 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.211836 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.211838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.211939 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.211942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.213014 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.213015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.214369 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.214373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.215271 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.215272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.215347 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.215350 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.215402 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.215431 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":51}
I20260810 15:23:19.215512 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.211436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:34585 (local address 127.4.64.66:45689)
0810 15:23:19.211658 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.211690 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.211738 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.211889 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.211890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.211891 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.211954 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.211955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.213015 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.213015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.214369 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.214373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.215202 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.215203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.215287 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.215292 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.215357 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.215410 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":39,"thread_start_us":124,"threads_started":1}
I20260810 15:23:19.215935 20486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1dc4b1cb114146af5c8a8e4c329737" candidate_uuid: "410d8070b5c8497da243188b8add0f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" is_pre_election: true
I20260810 15:23:19.215924 20804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1dc4b1cb114146af5c8a8e4c329737" candidate_uuid: "410d8070b5c8497da243188b8add0f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e555807f830a490a9a571637e90b9f57" is_pre_election: true
I20260810 15:23:19.216120 20486 raft_consensus.cc:2454] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 410d8070b5c8497da243188b8add0f6b in term 0.
I20260810 15:23:19.216128 20804 raft_consensus.cc:2454] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 410d8070b5c8497da243188b8add0f6b in term 0.
I20260810 15:23:19.216437 20570 leader_election.cc:302] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57; no voters:
I20260810 15:23:19.216596 20887 raft_consensus.cc:2781] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:23:19.216656 20887 raft_consensus.cc:482] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:23:19.216729 20887 raft_consensus.cc:3037] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:19.217479 20887 raft_consensus.cc:504] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.217625 20887 leader_election.cc:288] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [CANDIDATE]: Term 1 election: Requested vote from peers e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585), e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285)
I20260810 15:23:19.217805 20486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1dc4b1cb114146af5c8a8e4c329737" candidate_uuid: "410d8070b5c8497da243188b8add0f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
I20260810 15:23:19.217844 20804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1dc4b1cb114146af5c8a8e4c329737" candidate_uuid: "410d8070b5c8497da243188b8add0f6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e555807f830a490a9a571637e90b9f57"
I20260810 15:23:19.217901 20486 raft_consensus.cc:3037] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:19.217916 20804 raft_consensus.cc:3037] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:19.218818 20804 raft_consensus.cc:2454] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 410d8070b5c8497da243188b8add0f6b in term 1.
I20260810 15:23:19.218818 20486 raft_consensus.cc:2454] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 410d8070b5c8497da243188b8add0f6b in term 1.
I20260810 15:23:19.218978 20570 leader_election.cc:302] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57; no voters:
I20260810 15:23:19.219089 20887 raft_consensus.cc:2781] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:19.219287 20887 raft_consensus.cc:686] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 1 LEADER]: Becoming Leader. State: Replica: 410d8070b5c8497da243188b8add0f6b, State: Running, Role: LEADER
I20260810 15:23:19.219398 20887 consensus_queue.cc:237] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.220413 20321 catalog_manager.cc:5495] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 410d8070b5c8497da243188b8add0f6b (127.4.64.66). New cstate: current_term: 1 leader_uuid: "410d8070b5c8497da243188b8add0f6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } health_report { overall_health: HEALTHY } } }
I20260810 15:23:19.236644 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.234022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.4.64.66:38903)
0810 15:23:19.234118 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.234120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.234128 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:23:19.234183 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.234185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.234214 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.234323 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:19.234492 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.235006 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.235809 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.236011 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.236099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.236283 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:19.236404 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:23:19.236499 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.236522 (+ 23us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.236545 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 15:23:19.238992 20486 raft_consensus.cc:1261] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Refusing update from remote peer 410d8070b5c8497da243188b8add0f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:19.238992 20804 raft_consensus.cc:1261] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Refusing update from remote peer 410d8070b5c8497da243188b8add0f6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:19.239465 20893 consensus_queue.cc:1035] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:19.239614 20887 consensus_queue.cc:1035] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:19.242543 20900 mvcc.cc:205] Tried to move back new op lower bound from 7316993635279065088 to 7316993635203465216. Current Snapshot: MvccSnapshot[applied={T|T < 7316993635279065088}]
I20260810 15:23:19.244707 20321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:19.244818 20321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:19.245702 20321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Sending DeleteTablet for 3 replicas of tablet aa1dc4b1cb114146af5c8a8e4c329737
I20260810 15:23:19.246061 20464 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:19 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37998
I20260810 15:23:19.246089 20781 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:19 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:39860
I20260810 15:23:19.246323 20910 tablet_replica.cc:330] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: stopping tablet replica
I20260810 15:23:19.246376 20625 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:19 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37956
I20260810 15:23:19.246444 20910 raft_consensus.cc:2229] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:19.246511 20909 tablet_replica.cc:330] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: stopping tablet replica
I20260810 15:23:19.246554 20910 raft_consensus.cc:2258] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:19.246598 20909 raft_consensus.cc:2229] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:19.246732 20909 raft_consensus.cc:2258] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:19.246737 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 20390
I20260810 15:23:19.247361 20912 tablet_replica.cc:330] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: stopping tablet replica
I20260810 15:23:19.247452 20912 raft_consensus.cc:2229] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:23:19.247536 20912 raft_consensus.cc:2258] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:19.249212 20912 ts_tablet_manager.cc:1905] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:19.249862 20910 ts_tablet_manager.cc:1905] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:19.251236 20912 ts_tablet_manager.cc:1918] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:19.251324 20912 log.cc:1196] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/aa1dc4b1cb114146af5c8a8e4c329737
I20260810 15:23:19.251632 20912 ts_tablet_manager.cc:1939] T aa1dc4b1cb114146af5c8a8e4c329737 P 410d8070b5c8497da243188b8add0f6b: Deleting consensus metadata
I20260810 15:23:19.251778 20910 ts_tablet_manager.cc:1918] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:19.251871 20910 log.cc:1196] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/aa1dc4b1cb114146af5c8a8e4c329737
I20260810 15:23:19.252139 20303 catalog_manager.cc:4841] TS 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): tablet aa1dc4b1cb114146af5c8a8e4c329737 (table test-table [id=19abbf6f38e4497e9c7f73227e03009d]) successfully deleted
I20260810 15:23:19.252182 20910 ts_tablet_manager.cc:1939] T aa1dc4b1cb114146af5c8a8e4c329737 P e555807f830a490a9a571637e90b9f57: Deleting consensus metadata
I20260810 15:23:19.252658 20301 catalog_manager.cc:4841] TS e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): tablet aa1dc4b1cb114146af5c8a8e4c329737 (table test-table [id=19abbf6f38e4497e9c7f73227e03009d]) successfully deleted
W20260810 15:23:19.263851 20302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:34585 (error 108)
I20260810 15:23:19.264525 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.264369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:34585 (local address 127.0.0.1:38006)
0810 15:23:19.264435 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.264486 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:34585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 15:23:19.264564 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 20549
W20260810 15:23:19.267500 20302 catalog_manager.cc:4584] TS e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa1dc4b1cb114146af5c8a8e4c329737: Network error: Client connection negotiation failed: client connection to 127.4.64.65:34585: connect: Connection refused (error 111)
I20260810 15:23:19.271634 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 20713
I20260810 15:23:19.281539 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 20280
I20260810 15:23:19.290733 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44711
--webserver_interface=127.4.64.126
--webserver_port=37035
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:19.385875 20915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:19.386188 20915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:19.386235 20915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:19.387562 20915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:19.387656 20915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:19.387687 20915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:19.387719 20915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:19.387753 20915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:19.389489 20915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=37035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:19.389853 20915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:19.390130 20915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:19.400923 20922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.400923 20923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.400979 20925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:19.400982 20915 server_base.cc:1034] running on GCE node
I20260810 15:23:19.401669 20915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:19.402061 20915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:19.403268 20915 hybrid_clock.cc:648] HybridClock initialized: now 1786375399403250 us; error 48 us; skew 500 ppm
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:19.406388 20915 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:19.406957 20915 webserver.cc:460] Webserver started at http://127.4.64.126:37035/ using document root <none> and password file <none>
I20260810 15:23:19.407187 20915 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:19.407250 20915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:19.408787 20915 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:19.409672 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.409879 20915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:19.409984 20915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e94c519bfc7c42fe9d30a821b3b0b056"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "173bbd69bfa43056aeb13a5765be6fd5"
server_key_iv: "e15bf77151fbb9d5359941abca3099b8"
server_key_version: "encryptionkey@0"
I20260810 15:23:19.410386 20915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:19.435738 20915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:19.438457 20915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:19.438598 20915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:19.443174 20915 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44711
I20260810 15:23:19.443254 20999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44711 every 8 connection(s)
I20260810 15:23:19.443723 20915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:19.445827 21000 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:23:19.446583 21000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap starting.
I20260810 15:23:19.448726 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 20915
I20260810 15:23:19.449143 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:34585
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:44711
--webserver_port=39131
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:19.457190 21000 log.cc:824] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.459544 21000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:19.459919 21000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap complete.
I20260810 15:23:19.462672 21000 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:19.462931 21000 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94c519bfc7c42fe9d30a821b3b0b056, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.463126 21000 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:19.463202 21000 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:19.463266 21000 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:19.463320 21000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:23:19.464514 21000 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:19.464634 21000 leader_election.cc:302] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94c519bfc7c42fe9d30a821b3b0b056; no voters:
I20260810 15:23:19.464884 21000 leader_election.cc:288] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:23:19.464926 21005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:23:19.465058 21005 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 LEADER]: Becoming Leader. State: Replica: e94c519bfc7c42fe9d30a821b3b0b056, State: Running, Role: LEADER
I20260810 15:23:19.465183 21005 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:19.465220 21000 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:19.465490 21006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } } }
I20260810 15:23:19.465790 21006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:19.465490 21005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94c519bfc7c42fe9d30a821b3b0b056. Latest consensus state: current_term: 2 leader_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } } }
I20260810 15:23:19.466109 21005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:19.467352 21024 catalog_manager.cc:1259] Loaded cluster ID: d6660c12d8ea4949a28c7c6e3c152dcb
I20260810 15:23:19.467422 21024 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: loading cluster ID for follower catalog manager: success
W20260810 15:23:19.467636 21024 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260810 15:23:19.467700 21024 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260810 15:23:19.467758 21016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:19.468098 21016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:19.468205 21016 catalog_manager.cc:1259] Loaded cluster ID: d6660c12d8ea4949a28c7c6e3c152dcb
I20260810 15:23:19.468243 21016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:23:19.468408 21016 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 15:23:19.471572 21016 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 15:23:19.474897 21016 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:19.475131 21016 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Loaded TSK: 0
I20260810 15:23:19.475365 21016 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:23:19.555608 21002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:19.555914 21002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:19.555955 21002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:19.557124 21002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:19.557169 21002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:19.557195 21002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:19.558743 21002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:34585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=39131
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:19.558990 21002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:19.559254 21002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:19.564944 21034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.564945 21031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.564916 21032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:19.565274 21002 server_base.cc:1034] running on GCE node
I20260810 15:23:19.565485 21002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:19.565815 21002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:19.566987 21002 hybrid_clock.cc:648] HybridClock initialized: now 1786375399566970 us; error 34 us; skew 500 ppm
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:19.569931 21002 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:19.570508 21002 webserver.cc:460] Webserver started at http://127.4.64.65:39131/ using document root <none> and password file <none>
I20260810 15:23:19.570751 21002 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:19.570842 21002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:19.572310 21002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:23:19.573038 21043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.573235 21002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:19.573326 21002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "17d3da8ab7445a8044b8f82e0cd24dca"
server_key_iv: "2a1966fa1f715d94f4d51006b3a98f3b"
server_key_version: "encryptionkey@0"
I20260810 15:23:19.573753 21002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:19.594889 21002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:19.597529 21002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:19.597774 21002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:19.598073 21002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:19.598651 21050 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:23:19.600428 21002 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:23:19.600485 21002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:23:19.600523 21002 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:23:19.601266 21002 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:23:19.601316 21002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:23:19.601365 21050 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap starting.
I20260810 15:23:19.609756 21002 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:34585
I20260810 15:23:19.609941 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:34585 every 8 connection(s)
I20260810 15:23:19.610327 21002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:19.614593 21050 log.cc:824] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.616845 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21002
I20260810 15:23:19.617208 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:38903
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:44711
--webserver_port=43645
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:19.617209 21050 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:19.617565 21050 tablet_bootstrap.cc:492] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap complete.
I20260810 15:23:19.619011 21050 ts_tablet_manager.cc:1397] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20260810 15:23:19.620357 21050 raft_consensus.cc:348] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:19.620628 21050 raft_consensus.cc:729] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822f28d3a7741a3b8eb56e3aee2c460, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.620816 21050 consensus_queue.cc:260] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } } peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } }
I20260810 15:23:19.621119 21050 ts_tablet_manager.cc:1428] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 15:23:19.628515 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.611129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:53713)
0810 15:23:19.611373 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.611423 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.612081 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.613152 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.613157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.613791 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:19.614144 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.614154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.615883 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.615884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.617118 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.617121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.618420 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.618422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.618573 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.619220 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:19.619230 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:19.621259 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:19.626451 (+ 5192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.626461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.626464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.626812 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.627162 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.627163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.627163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.627339 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.627948 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:19.627955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:19.628082 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.628194 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.628363 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":141,"threads_started":1}
I20260810 15:23:19.628707 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.611295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:53713 (local address 127.4.64.126:44711)
0810 15:23:19.612656 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.612667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.612713 (+ 46us) server_negotiation.cc:407] Connection header received
0810 15:23:19.612860 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.612866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.612922 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.613064 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.614338 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.615744 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.617520 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.618286 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.618881 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.621363 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.621367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.621368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.621382 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.626321 (+ 4939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.626919 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.626924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.626924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.627067 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.627423 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.627424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.627425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.627845 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.627949 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.628280 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.628450 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":95,"threads_started":1}
I20260810 15:23:19.629377 21180 heartbeater.cc:347] Connected to a master server at 127.4.64.126:44711
I20260810 15:23:19.629498 21180 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:19.629755 21180 heartbeater.cc:511] Master 127.4.64.126:44711 requested a full tablet report, sending...
I20260810 15:23:19.630913 20951 ts_manager.cc:194] Registered new tserver with Master: e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585)
I20260810 15:23:19.632282 20951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:53713
I20260810 15:23:19.633145 21180 heartbeater.cc:503] Master 127.4.64.126:44711 was elected leader, sending a full tablet report...
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:23:19.640748 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.631594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:34585 (local address 127.0.0.1:38022)
0810 15:23:19.631885 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.631919 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.632013 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.632204 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.632210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.632610 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:19.632672 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.632675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.634102 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.634103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.634704 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.634707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.635360 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.635361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.635436 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.636027 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:19.636032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:19.637477 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:19.639526 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.639532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.639534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.639747 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.640017 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.640018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.640018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.640086 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.640420 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:19.640426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:19.640478 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.640553 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.640658 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260810 15:23:19.640841 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.631654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.4.64.65:34585)
0810 15:23:19.631808 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.631812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.631994 (+ 182us) server_negotiation.cc:407] Connection header received
0810 15:23:19.632091 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.632094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.632110 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.632201 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.632777 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.634021 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.634789 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.635251 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.635326 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.637568 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.637577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.637578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.637586 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.639422 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.639841 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.639845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.639845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.639915 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.640156 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.640157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.640157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.640346 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.640427 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.640608 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.640741 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260810 15:23:19.641309 21101 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1dc4b1cb114146af5c8a8e4c329737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:19 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:38022
I20260810 15:23:19.641917 21191 tablet_replica.cc:330] stopping tablet replica
I20260810 15:23:19.642032 21191 raft_consensus.cc:2229] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:19.642107 21191 raft_consensus.cc:2258] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:19.642411 21191 ts_tablet_manager.cc:1905] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:19.644711 21191 ts_tablet_manager.cc:1918] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:19.644783 21191 log.cc:1196] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/aa1dc4b1cb114146af5c8a8e4c329737
I20260810 15:23:19.645020 21191 ts_tablet_manager.cc:1939] T aa1dc4b1cb114146af5c8a8e4c329737 P e822f28d3a7741a3b8eb56e3aee2c460: Deleting consensus metadata
I20260810 15:23:19.645569 20935 catalog_manager.cc:4841] TS e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585): tablet aa1dc4b1cb114146af5c8a8e4c329737 (table test-table [id=19abbf6f38e4497e9c7f73227e03009d]) successfully deleted
W20260810 15:23:19.711026 21187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:19.711344 21187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:19.711395 21187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:19.712774 21187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:19.712860 21187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:19.712910 21187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:19.714725 21187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:38903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=43645
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:19.715039 21187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:19.715338 21187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:19.721318 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.721323 21197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.721345 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:19.721668 21187 server_base.cc:1034] running on GCE node
I20260810 15:23:19.721901 21187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:19.722194 21187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:19.723428 21187 hybrid_clock.cc:648] HybridClock initialized: now 1786375399723418 us; error 21 us; skew 500 ppm
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:19.726519 21187 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:19.727094 21187 webserver.cc:460] Webserver started at http://127.4.64.66:43645/ using document root <none> and password file <none>
I20260810 15:23:19.727382 21187 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:19.727470 21187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:19.729108 21187 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:23:19.729863 21208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.730067 21187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:19.730158 21187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "410d8070b5c8497da243188b8add0f6b"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "a2355339f56ae99d66176df25995ff1e"
server_key_iv: "b679fb5c9b772af332a4ccd9c48ac24a"
server_key_version: "encryptionkey@0"
I20260810 15:23:19.730511 21187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:19.746366 21187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:19.748539 21187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:19.748665 21187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:19.748873 21187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:19.749208 21187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:19.749246 21187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.749276 21187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:19.749290 21187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.755715 21187 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38903
I20260810 15:23:19.755810 21339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38903 every 8 connection(s)
I20260810 15:23:19.756191 21187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:19.765643 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21187
I20260810 15:23:19.765978 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:44285
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:44711
--webserver_port=36785
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:19.767361 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.757051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:36433 (local address 127.4.64.126:44711)
0810 15:23:19.757135 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.757137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.757490 (+ 353us) server_negotiation.cc:407] Connection header received
0810 15:23:19.758081 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.758082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.758093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.758153 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.759160 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.759678 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.761022 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.761224 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.761283 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.763833 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.763835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.763835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.763843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.765536 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.766228 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.766230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.766230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.766299 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.766572 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.766573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.766574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.766734 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.766805 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.767113 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.767217 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 15:23:19.767499 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.757019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:36433)
0810 15:23:19.757350 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.757401 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.757982 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.758227 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.758230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.758847 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:19.759061 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.759065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.759832 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.759833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.760928 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.760932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.761315 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.761316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.761448 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.762071 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:19.762081 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:19.763727 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:19.765691 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.765700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.765702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.766032 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.766364 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.766364 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.766365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.766491 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.766826 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:19.766833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:19.766965 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.767055 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.767269 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260810 15:23:19.768096 21340 heartbeater.cc:347] Connected to a master server at 127.4.64.126:44711
I20260810 15:23:19.768231 21340 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:19.768428 21340 heartbeater.cc:511] Master 127.4.64.126:44711 requested a full tablet report, sending...
I20260810 15:23:19.768903 20951 ts_manager.cc:194] Registered new tserver with Master: 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
I20260810 15:23:19.769299 20951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:36433
W20260810 15:23:19.853531 21343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:19.853844 21343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:19.853890 21343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:19.855098 21343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:19.855182 21343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:19.855228 21343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:19.856931 21343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:44285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=36785
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:19.857239 21343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:19.857503 21343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:19.863318 21349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.863384 21352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:19.863319 21350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:19.863543 21343 server_base.cc:1034] running on GCE node
I20260810 15:23:19.863803 21343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:19.864089 21343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:19.865291 21343 hybrid_clock.cc:648] HybridClock initialized: now 1786375399865248 us; error 40 us; skew 500 ppm
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:19.868608 21343 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:19.869208 21343 webserver.cc:460] Webserver started at http://127.4.64.67:36785/ using document root <none> and password file <none>
I20260810 15:23:19.869472 21343 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:19.869549 21343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:19.871297 21343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:19.872062 21360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.872376 21343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:19.872504 21343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "e555807f830a490a9a571637e90b9f57"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "cfb046667d05aa7a015828bf37dd07b3"
server_key_iv: "30341a270a4dc6456993b4788deca67a"
server_key_version: "encryptionkey@0"
I20260810 15:23:19.872911 21343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:19.885210 21343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:19.887639 21343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:19.887847 21343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:19.888125 21343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:19.888486 21343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:19.888545 21343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.888592 21343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:19.888620 21343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:19.895181 21343 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:44285
I20260810 15:23:19.895241 21507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:44285 every 8 connection(s)
I20260810 15:23:19.895704 21343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:23:19 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375399, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:19.903469 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21343
I20260810 15:23:19.907177 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.896574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:58819 (local address 127.4.64.126:44711)
0810 15:23:19.896651 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.896654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.896905 (+ 251us) server_negotiation.cc:407] Connection header received
0810 15:23:19.897524 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.897525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.897538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.897591 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.898679 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.899317 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.900547 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.900751 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.900816 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.903633 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.903635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.903635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.903645 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.905496 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.906068 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.906069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.906069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.906130 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.906428 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.906428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.906429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.906534 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.906584 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.906974 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.907074 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:23:19.907323 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.896565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:58819)
0810 15:23:19.896769 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.896811 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.897411 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.897686 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.897689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.898318 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:19.898548 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.898552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.899458 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.899459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.900445 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.900448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.900855 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.900856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.900988 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.901685 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:19.901696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:19.903488 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:19.905584 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.905595 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.905598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.905999 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.906198 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:19.906198 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:19.906199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:19.906355 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:19.906596 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:19.906603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:19.906789 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.906905 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.907094 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260810 15:23:19.907886 21508 heartbeater.cc:347] Connected to a master server at 127.4.64.126:44711
I20260810 15:23:19.908025 21508 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:19.908263 21508 heartbeater.cc:511] Master 127.4.64.126:44711 requested a full tablet report, sending...
I20260810 15:23:19.908704 20951 ts_manager.cc:194] Registered new tserver with Master: e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285)
I20260810 15:23:19.909175 20951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:58819
I20260810 15:23:19.911698 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.903967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.4.64.126:44711)
0810 15:23:19.904119 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.904121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.904305 (+ 184us) server_negotiation.cc:407] Connection header received
0810 15:23:19.904352 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.904353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.904363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.904422 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:19.905214 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.905862 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.906951 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.907257 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.907329 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.908991 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:19.908992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:19.908993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:19.908999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:19.910616 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.910990 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.910991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.910991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.911045 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:19.911231 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:19.911232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:19.911232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:19.911356 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:19.911477 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.911564 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.911630 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260810 15:23:19.912654 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:23:19.915740 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.913077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.4.64.126:44711)
0810 15:23:19.913241 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.913243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.913252 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:23:19.913300 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.913301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.913334 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.913404 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:19.913489 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.914117 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.914908 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.915232 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.915307 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.915463 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:19.915572 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:23:19.915644 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.915660 (+ 16us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.915682 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260810 15:23:19.916082 20951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:23:19.917991 20951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:23:19.920492 21101 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f (DEFAULT_TABLE table=test-table [id=e366dbb872194ff8860c76d66f749aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:19.920724 21101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:19.922236 21050 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap starting.
I20260810 15:23:19.922610 21050 tablet_bootstrap.cc:654] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:19.923475 21050 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: No bootstrap required, opened a new log
I20260810 15:23:19.923528 21050 ts_tablet_manager.cc:1397] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 15:23:19.923909 21050 raft_consensus.cc:348] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:19.924013 21050 raft_consensus.cc:374] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:19.924067 21050 raft_consensus.cc:729] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822f28d3a7741a3b8eb56e3aee2c460, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.924121 21050 consensus_queue.cc:260] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:19.924234 21050 ts_tablet_manager.cc:1428] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 15:23:19.924321 21189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.920081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38903 (local address 127.0.0.1:37982)
0810 15:23:19.920139 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.920151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.920190 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.920631 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.920633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.920635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.920694 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.920696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.921902 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.921904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.922984 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.922987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.923983 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.923984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.924088 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.924113 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.924174 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.924229 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 15:23:19.924630 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.920229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.4.64.66:38903)
0810 15:23:19.920432 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.920437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.920448 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:23:19.920489 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.920491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.920507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.920596 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.920816 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.921800 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.923083 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.923886 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.923967 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.924029 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.924491 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.924537 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
I20260810 15:23:19.925419 21265 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f (DEFAULT_TABLE table=test-table [id=e366dbb872194ff8860c76d66f749aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:19.925906 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:19.926075 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.920081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:44285 (local address 127.0.0.1:39864)
0810 15:23:19.920236 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:19.920245 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:19.920283 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:19.922519 (+ 2236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:19.922521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:19.922524 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:19.922568 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.922570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.923796 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.923797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.924854 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:19.924857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.925824 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.925825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:19.925892 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.925896 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:23:19.925953 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:23:19.925996 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
I20260810 15:23:19.926086 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:19.920492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.4.64.67:44285)
0810 15:23:19.922336 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:19.922341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:19.922353 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:23:19.922393 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:19.922395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:19.922418 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:19.922507 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:19.922625 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.923643 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.925106 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:19.925738 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:19.925809 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:19.925873 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:19.925968 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 15:23:19.926008 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1797,"thread_start_us":104,"threads_started":1}
I20260810 15:23:19.926872 21420 tablet_service.cc:1467] Processing CreateTablet for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f (DEFAULT_TABLE table=test-table [id=e366dbb872194ff8860c76d66f749aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:19.927191 21420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:19.928989 21518 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Bootstrap starting.
I20260810 15:23:19.929760 21518 tablet_bootstrap.cc:654] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:19.930145 21519 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Bootstrap starting.
I20260810 15:23:19.930384 21518 log.cc:824] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.930841 21519 tablet_bootstrap.cc:654] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:19.931349 21518 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: No bootstrap required, opened a new log
I20260810 15:23:19.931433 21518 ts_tablet_manager.cc:1397] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:23:19.931445 21519 log.cc:824] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:19.932298 21519 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: No bootstrap required, opened a new log
I20260810 15:23:19.932468 21519 ts_tablet_manager.cc:1397] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:19.933535 21518 raft_consensus.cc:348] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:19.933749 21518 raft_consensus.cc:374] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:19.933791 21518 raft_consensus.cc:729] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410d8070b5c8497da243188b8add0f6b, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.933920 21518 consensus_queue.cc:260] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:19.934227 21518 ts_tablet_manager.cc:1428] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 15:23:19.934370 21340 heartbeater.cc:503] Master 127.4.64.126:44711 was elected leader, sending a full tablet report...
I20260810 15:23:19.934621 21519 raft_consensus.cc:348] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:19.934804 21519 raft_consensus.cc:374] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:19.934845 21519 raft_consensus.cc:729] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e555807f830a490a9a571637e90b9f57, State: Initialized, Role: FOLLOWER
I20260810 15:23:19.934954 21519 consensus_queue.cc:260] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:19.935248 21519 ts_tablet_manager.cc:1428] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:23:19.935302 21508 heartbeater.cc:503] Master 127.4.64.126:44711 was elected leader, sending a full tablet report...
W20260810 15:23:20.006888 21341 tablet.cc:2369] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:23:20.111356 21182 tablet.cc:2369] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:23:20.146391 21509 tablet.cc:2369] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:20.171051 21523 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:20.171241 21523 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:20.171655 21523 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903), e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585)
I20260810 15:23:20.176151 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.171933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38903 (local address 127.4.64.67:35245)
0810 15:23:20.172114 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.172124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:20.172188 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:20.172349 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:20.172349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:20.172351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:20.172453 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:20.172456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.173507 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.173507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:20.174933 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:20.174935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.175841 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.175843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:20.175916 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:20.175920 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:23:20.175998 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 15:23:20.176050 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":55,"thread_start_us":118,"threads_started":1}
I20260810 15:23:20.176177 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.171989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:35245 (local address 127.4.64.66:38903)
0810 15:23:20.172060 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:20.172063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:20.172208 (+ 145us) server_negotiation.cc:407] Connection header received
0810 15:23:20.172248 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:20.172250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:20.172269 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:20.172344 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:20.172562 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.173406 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.175048 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.175715 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.175806 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:20.175821 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:20.176026 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 15:23:20.176098 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:23:20.176403 21525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.172006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:47417 (local address 127.4.64.65:34585)
0810 15:23:20.172252 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:20.172254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:20.172264 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:23:20.172294 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:20.172295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:20.172326 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:20.172414 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:20.172562 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.173483 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.175048 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.175881 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.175952 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:20.176234 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:20.176260 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 15:23:20.176323 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":59,"threads_started":1}
I20260810 15:23:20.176546 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.171929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:34585 (local address 127.4.64.67:47417)
0810 15:23:20.172110 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.172123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:20.172188 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:20.172433 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:20.172433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:20.172435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:20.172480 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:20.172481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.173570 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.173570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:20.174927 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:20.174931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.175964 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.175965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:20.176035 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:20.176037 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:23:20.176077 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 15:23:20.176177 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":85}
I20260810 15:23:20.176597 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebecb8d7b9d4fea98a45a72dfe31f2f" candidate_uuid: "e555807f830a490a9a571637e90b9f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410d8070b5c8497da243188b8add0f6b" is_pre_election: true
I20260810 15:23:20.176801 21290 raft_consensus.cc:2454] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e555807f830a490a9a571637e90b9f57 in term 0.
I20260810 15:23:20.176847 21127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebecb8d7b9d4fea98a45a72dfe31f2f" candidate_uuid: "e555807f830a490a9a571637e90b9f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" is_pre_election: true
I20260810 15:23:20.177021 21127 raft_consensus.cc:2454] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e555807f830a490a9a571637e90b9f57 in term 0.
I20260810 15:23:20.177107 21363 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57; no voters:
I20260810 15:23:20.177353 21523 raft_consensus.cc:2781] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:23:20.177436 21523 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:23:20.177474 21523 raft_consensus.cc:3037] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:20.178289 21523 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:20.178454 21523 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [CANDIDATE]: Term 1 election: Requested vote from peers 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903), e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585)
I20260810 15:23:20.178639 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebecb8d7b9d4fea98a45a72dfe31f2f" candidate_uuid: "e555807f830a490a9a571637e90b9f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410d8070b5c8497da243188b8add0f6b"
I20260810 15:23:20.178743 21290 raft_consensus.cc:3037] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:20.179508 21290 raft_consensus.cc:2454] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e555807f830a490a9a571637e90b9f57 in term 1.
I20260810 15:23:20.179708 21127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ebecb8d7b9d4fea98a45a72dfe31f2f" candidate_uuid: "e555807f830a490a9a571637e90b9f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
I20260810 15:23:20.179777 21363 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57; no voters:
I20260810 15:23:20.179836 21127 raft_consensus.cc:3037] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:20.179878 21523 raft_consensus.cc:2781] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:20.180060 21523 raft_consensus.cc:686] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 1 LEADER]: Becoming Leader. State: Replica: e555807f830a490a9a571637e90b9f57, State: Running, Role: LEADER
I20260810 15:23:20.180172 21523 consensus_queue.cc:237] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:20.180732 21127 raft_consensus.cc:2454] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e555807f830a490a9a571637e90b9f57 in term 1.
I20260810 15:23:20.181066 20951 catalog_manager.cc:5495] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 reported cstate change: term changed from 0 to 1, leader changed from <none> to e555807f830a490a9a571637e90b9f57 (127.4.64.67). New cstate: current_term: 1 leader_uuid: "e555807f830a490a9a571637e90b9f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } health_report { overall_health: UNKNOWN } } }
I20260810 15:23:20.214378 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.210893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.4.64.67:44285)
0810 15:23:20.210968 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:20.210970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:20.211019 (+ 49us) server_negotiation.cc:407] Connection header received
0810 15:23:20.211296 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:20.211298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:20.211324 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:20.211408 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:20.211572 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.212212 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.213024 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:20.213436 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:20.213559 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:20.213771 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:20.213995 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:23:20.214108 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:20.214145 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:23:20.214196 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:23:20.217083 21290 raft_consensus.cc:1261] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Refusing update from remote peer e555807f830a490a9a571637e90b9f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:20.217195 21127 raft_consensus.cc:1261] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Refusing update from remote peer e555807f830a490a9a571637e90b9f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:20.217681 21523 consensus_queue.cc:1035] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:20.217947 21526 consensus_queue.cc:1035] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:20.220386 21531 mvcc.cc:205] Tried to move back new op lower bound from 7316993639284727808 to 7316993639138586624. Current Snapshot: MvccSnapshot[applied={T|T < 7316993639284727808}]
I20260810 15:23:20.221102 21532 mvcc.cc:205] Tried to move back new op lower bound from 7316993639284727808 to 7316993639138586624. Current Snapshot: MvccSnapshot[applied={T|T < 7316993639284727808}]
I20260810 15:23:20.223567 20949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:20.223706 20949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:20.224617 20949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Sending DeleteTablet for 3 replicas of tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:20.224982 21265 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:20 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:37982
I20260810 15:23:20.225037 21420 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:20 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:39864
I20260810 15:23:20.225061 21101 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:20 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:38022
I20260810 15:23:20.225201 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 21002
I20260810 15:23:20.225298 21542 tablet_replica.cc:330] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: stopping tablet replica
I20260810 15:23:20.225410 21542 raft_consensus.cc:2229] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:20.225520 21542 raft_consensus.cc:2258] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:20.230002 21545 tablet_replica.cc:330] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: stopping tablet replica
I20260810 15:23:20.230212 21545 raft_consensus.cc:2229] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:23:20.230448 21545 raft_consensus.cc:2258] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:20.231843 21545 ts_tablet_manager.cc:1905] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:20.233726 21545 ts_tablet_manager.cc:1918] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:20.233995 21545 log.cc:1196] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:20.234331 21542 ts_tablet_manager.cc:1905] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:20.234405 21545 ts_tablet_manager.cc:1939] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e555807f830a490a9a571637e90b9f57: Deleting consensus metadata
I20260810 15:23:20.234907 20934 catalog_manager.cc:4841] TS e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f (table test-table [id=e366dbb872194ff8860c76d66f749aa6]) successfully deleted
I20260810 15:23:20.236315 21542 ts_tablet_manager.cc:1918] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:20.236421 21542 log.cc:1196] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:20.236698 21542 ts_tablet_manager.cc:1939] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P 410d8070b5c8497da243188b8add0f6b: Deleting consensus metadata
I20260810 15:23:20.237241 20937 catalog_manager.cc:4841] TS 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f (table test-table [id=e366dbb872194ff8860c76d66f749aa6]) successfully deleted
W20260810 15:23:20.237502 20935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:34585 (error 108)
W20260810 15:23:20.238188 21514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:23:20.237987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:34585 (local address 127.0.0.1:38030)
0810 15:23:20.238081 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.238121 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:34585: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 15:23:20.238392 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 21187
W20260810 15:23:20.238341 20935 catalog_manager.cc:4584] TS e822f28d3a7741a3b8eb56e3aee2c460 (127.4.64.65:34585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ebecb8d7b9d4fea98a45a72dfe31f2f: Network error: Client connection negotiation failed: client connection to 127.4.64.65:34585: connect: Connection reset by peer (error 104)
I20260810 15:23:20.252655 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 21343
I20260810 15:23:20.262781 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 20915
I20260810 15:23:20.272567 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44711
--webserver_interface=127.4.64.126
--webserver_port=37035
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:20.359313 21547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:20.359529 21547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:20.359552 21547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:20.360745 21547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:20.360792 21547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:20.360806 21547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:20.360819 21547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:20.360879 21547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:20.362489 21547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=37035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:20.362782 21547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:20.363040 21547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:20.368507 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.368626 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.368556 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:20.368515 21547 server_base.cc:1034] running on GCE node
I20260810 15:23:20.368995 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:20.369367 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:20.370570 21547 hybrid_clock.cc:648] HybridClock initialized: now 1786375400370560 us; error 52 us; skew 500 ppm
Aug 10 15:23:20 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:20.373870 21547 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:20.374356 21547 webserver.cc:460] Webserver started at http://127.4.64.126:37035/ using document root <none> and password file <none>
I20260810 15:23:20.374620 21547 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:20.374696 21547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:20.376348 21547 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:23:20.377209 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.377426 21547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:20.377523 21547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e94c519bfc7c42fe9d30a821b3b0b056"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "173bbd69bfa43056aeb13a5765be6fd5"
server_key_iv: "e15bf77151fbb9d5359941abca3099b8"
server_key_version: "encryptionkey@0"
I20260810 15:23:20.377990 21547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:20.393795 21547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:20.396430 21547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:20.396611 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:20.401022 21547 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44711
I20260810 15:23:20.401108 21629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44711 every 8 connection(s)
I20260810 15:23:20.401542 21547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:20.403420 21630 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:23:20.404047 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap starting.
I20260810 15:23:20.409039 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21547
I20260810 15:23:20.409401 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:34585
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:44711
--webserver_port=39131
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:20.414474 21630 log.cc:824] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:20.416870 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:20.417260 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056: Bootstrap complete.
I20260810 15:23:20.419814 21630 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:20.420056 21630 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94c519bfc7c42fe9d30a821b3b0b056, State: Initialized, Role: FOLLOWER
I20260810 15:23:20.420236 21630 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:20.420317 21630 raft_consensus.cc:388] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:20.420385 21630 raft_consensus.cc:482] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:20.420434 21630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:23:20.421561 21630 raft_consensus.cc:504] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:20.421739 21630 leader_election.cc:302] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94c519bfc7c42fe9d30a821b3b0b056; no voters:
I20260810 15:23:20.421975 21630 leader_election.cc:288] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 15:23:20.422020 21634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:23:20.422153 21634 raft_consensus.cc:686] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [term 3 LEADER]: Becoming Leader. State: Replica: e94c519bfc7c42fe9d30a821b3b0b056, State: Running, Role: LEADER
I20260810 15:23:20.422288 21630 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:20.422268 21634 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } }
I20260810 15:23:20.422585 21634 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94c519bfc7c42fe9d30a821b3b0b056. Latest consensus state: current_term: 3 leader_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } } }
I20260810 15:23:20.422683 21634 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:20.422948 21635 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94c519bfc7c42fe9d30a821b3b0b056" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44711 } } }
I20260810 15:23:20.423094 21635 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94c519bfc7c42fe9d30a821b3b0b056 [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:20.423233 21639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:20.424378 21639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:20.424563 21639 catalog_manager.cc:1259] Loaded cluster ID: d6660c12d8ea4949a28c7c6e3c152dcb
I20260810 15:23:20.424793 21639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 15:23:20.425135 21639 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7f905239ad google::LogMessage::Fail() at ??:0
@ 0x7f7f9052688a google::LogMessage::SendToLog() at ??:0
@ 0x7f7f905234de google::LogMessage::Flush() at ??:0
@ 0x7f7f90524a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7f984440f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7f9844d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7f923f10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7f923e6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7f9324a6db start_thread at ??:0
@ 0x7f7f8f02161f clone at ??:0
W20260810 15:23:20.519353 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:20.519572 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:20.519598 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:20.520815 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:20.520869 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:20.520898 21632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:20.522516 21632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:34585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=39131
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:20.522794 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:20.523077 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:20.528932 21675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.528891 21673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:20.529253 21632 server_base.cc:1034] running on GCE node
W20260810 15:23:20.528891 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:20.529558 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:20.529937 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:20.531136 21632 hybrid_clock.cc:648] HybridClock initialized: now 1786375400531108 us; error 39 us; skew 500 ppm
Aug 10 15:23:20 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:20.534119 21632 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:20.534709 21632 webserver.cc:460] Webserver started at http://127.4.64.65:39131/ using document root <none> and password file <none>
I20260810 15:23:20.534993 21632 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:20.535041 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:20.536536 21632 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:23:20.537302 21682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.537513 21632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:20.537657 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "e822f28d3a7741a3b8eb56e3aee2c460"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "17d3da8ab7445a8044b8f82e0cd24dca"
server_key_iv: "2a1966fa1f715d94f4d51006b3a98f3b"
server_key_version: "encryptionkey@0"
I20260810 15:23:20.538051 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:20.547133 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:20.549723 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:20.549888 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:20.550195 21632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:20.550719 21690 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:23:20.552253 21632 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:23:20.552327 21632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:23:20.552356 21632 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:23:20.553092 21632 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:23:20.553140 21632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:23:20.553186 21690 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap starting.
I20260810 15:23:20.561523 21632 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:34585
I20260810 15:23:20.561883 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:34585 every 8 connection(s)
I20260810 15:23:20.562389 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:20.566094 21690 log.cc:824] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:20.566625 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21632
I20260810 15:23:20.566900 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:38903
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:44711
--webserver_port=43645
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:20.570086 21690 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:20.570546 21690 tablet_bootstrap.cc:492] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Bootstrap complete.
I20260810 15:23:20.572070 21690 ts_tablet_manager.cc:1397] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260810 15:23:20.573761 21690 raft_consensus.cc:348] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:20.574266 21690 raft_consensus.cc:729] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822f28d3a7741a3b8eb56e3aee2c460, State: Initialized, Role: FOLLOWER
I20260810 15:23:20.574498 21690 consensus_queue.cc:260] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:20.574986 21690 ts_tablet_manager.cc:1428] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260810 15:23:20.656080 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:20.656363 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:20.656407 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:20.658883 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:20.658958 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:20.659013 21823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:20.660753 21823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:38903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=43645
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:20.661046 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:20.661324 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:20.664500 21821 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:23:20.563214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:39419)
0810 15:23:20.563403 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.563448 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:20.564228 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:20.664127 (+ 99899us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
W20260810 15:23:20.664840 21819 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:44711 (0 consecutive failures): Network error: Failed to ping master at 127.4.64.126:44711: Client connection negotiation failed: client connection to 127.4.64.126:44711: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:23:20.665225 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.665080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:57613)
0810 15:23:20.665154 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.665200 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 15:23:20.665645 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.665521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:33459)
0810 15:23:20.665568 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.665595 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 15:23:20.665793 21819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:23:20.668211 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.668148 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.668148 21832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:20.668433 21823 server_base.cc:1034] running on GCE node
I20260810 15:23:20.668646 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:20.668906 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:20.670302 21823 hybrid_clock.cc:648] HybridClock initialized: now 1786375400670285 us; error 28 us; skew 500 ppm
Aug 10 15:23:20 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:20.673897 21823 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:20.674386 21823 webserver.cc:460] Webserver started at http://127.4.64.66:43645/ using document root <none> and password file <none>
I20260810 15:23:20.674633 21823 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:20.674706 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:20.676196 21823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:20.676957 21843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.677183 21823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:20.677290 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "410d8070b5c8497da243188b8add0f6b"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "a2355339f56ae99d66176df25995ff1e"
server_key_iv: "b679fb5c9b772af332a4ccd9c48ac24a"
server_key_version: "encryptionkey@0"
I20260810 15:23:20.677734 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:20.689258 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:20.691830 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:20.692015 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:20.692320 21823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:20.692674 21823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:20.692729 21823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.692781 21823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:20.692828 21823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.699805 21823 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38903
I20260810 15:23:20.699863 21974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38903 every 8 connection(s)
I20260810 15:23:20.700280 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:23:20.701448 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.701028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:39869)
0810 15:23:20.701213 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.701293 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
W20260810 15:23:20.701917 21976 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:44711 (0 consecutive failures): Network error: Failed to ping master at 127.4.64.126:44711: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
I20260810 15:23:20.702277 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.702145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:59903)
0810 15:23:20.702220 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.702253 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 15:23:20.702581 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.702525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:42473)
0810 15:23:20.702546 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.702567 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260810 15:23:20.702737 21976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:23:20.703760 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21823
I20260810 15:23:20.704069 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:44285
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:44711
--webserver_port=36785
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:20.792994 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:20.793232 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:20.793259 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:20.794426 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:20.794468 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:20.794495 21979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:20.795943 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:44285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=36785
--tserver_master_addrs=127.4.64.126:44711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:20.796166 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:20.796368 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:20.802239 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.802239 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:20.802433 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:20.802551 21979 server_base.cc:1034] running on GCE node
I20260810 15:23:20.802804 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:20.803066 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:20.804322 21979 hybrid_clock.cc:648] HybridClock initialized: now 1786375400804306 us; error 28 us; skew 500 ppm
Aug 10 15:23:20 dist-test-slave-9zdj krb5kdc[20263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:20.807369 21979 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:20.807921 21979 webserver.cc:460] Webserver started at http://127.4.64.67:36785/ using document root <none> and password file <none>
I20260810 15:23:20.808197 21979 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:20.808249 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:20.809809 21979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:23:20.810564 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.810766 21979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:20.810858 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "e555807f830a490a9a571637e90b9f57"
format_stamp: "Formatted at 2026-08-10 15:23:18 on dist-test-slave-9zdj"
server_key: "cfb046667d05aa7a015828bf37dd07b3"
server_key_iv: "30341a270a4dc6456993b4788deca67a"
server_key_version: "encryptionkey@0"
I20260810 15:23:20.811233 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:20.825809 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:20.828234 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:20.828357 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:20.828562 21979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:20.828909 21979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:20.828946 21979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.828970 21979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:20.829020 21979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:20.835392 21979 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:44285
I20260810 15:23:20.835461 22130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:44285 every 8 connection(s)
I20260810 15:23:20.835832 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:23:20.836962 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.836549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:48199)
0810 15:23:20.836728 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.836814 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
W20260810 15:23:20.837746 22131 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:44711 (0 consecutive failures): Network error: Failed to ping master at 127.4.64.126:44711: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
I20260810 15:23:20.838201 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.838064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:52477)
0810 15:23:20.838140 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.838175 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 15:23:20.838575 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:20.838486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:57023)
0810 15:23:20.838526 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:20.838555 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:23:20.838781 22131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:23:20.841113 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 21979
I20260810 15:23:21.666431 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.666165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:36351)
0810 15:23:21.666312 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:21.666351 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260810 15:23:21.703395 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.703122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:49543)
0810 15:23:21.703294 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:21.703334 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260810 15:23:21.778201 22138 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:21.778347 22138 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:21.778792 22138 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285), 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
Aug 10 15:23:21 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.67@KRBTEST.COM
Aug 10 15:23:21 dist-test-slave-9zdj krb5kdc[20263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.66@KRBTEST.COM
I20260810 15:23:21.792359 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.779152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:33965 (local address 127.4.64.67:44285)
0810 15:23:21.779448 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:21.779457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:21.779477 (+ 20us) server_negotiation.cc:407] Connection header received
0810 15:23:21.780026 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:21.780029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:21.780124 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:21.780252 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:21.781389 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.782897 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.784599 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.785264 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.785390 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:21.788436 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:21.788441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:21.788441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:21.788455 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:21.790545 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:21.791067 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:21.791071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:21.791071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:21.791175 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:21.791446 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:21.791448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:21.791448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:21.791719 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:21.791833 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:21.792116 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 15:23:21.792251 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":68,"threads_started":1}
I20260810 15:23:21.792367 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.779020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:44285 (local address 127.4.64.65:33965)
0810 15:23:21.779321 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:21.779332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:21.779384 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:21.780331 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:21.780333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:21.780875 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:21.781182 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:21.781190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.783041 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.783042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:21.784473 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:21.784476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.785396 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.785397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:21.785561 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:21.786307 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:21.786319 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:21.788304 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:21.790639 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:21.790648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:21.790651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:21.790975 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:21.791250 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:21.791251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:21.791251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:21.791374 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:21.791817 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:21.791818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:21.791945 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 15:23:21.792053 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:23:21.792263 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":134,"thread_start_us":55,"threads_started":1}
I20260810 15:23:21.792410 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.779121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:54291 (local address 127.4.64.66:38903)
0810 15:23:21.779385 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:21.779398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:21.779439 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:23:21.780026 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:21.780029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:21.780094 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:21.780252 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:21.781353 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.782830 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.784600 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.785265 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.785390 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:21.788779 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:21.788783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:21.788784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:21.788795 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:21.790649 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:21.791109 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:21.791112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:21.791112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:21.791216 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:21.791470 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:21.791471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:21.791472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:21.791722 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:21.791823 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:21.792122 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 15:23:21.792280 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":58,"threads_started":1}
I20260810 15:23:21.792819 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.779012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38903 (local address 127.4.64.65:54291)
0810 15:23:21.779251 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:21.779265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:21.779332 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:21.780329 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:21.780333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:21.780875 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:21.781182 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:21.781190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.783010 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.783011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:21.784472 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:21.784476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:21.785394 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:21.785396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:21.785561 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:21.786340 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:21.786344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:21.788651 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:21.790744 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:21.790745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:21.790746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:21.791006 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:21.791287 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:21.791288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:21.791288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:21.791395 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:21.791806 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:21.791814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:21.791945 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 15:23:21.792052 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 15:23:21.792359 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":183}
W20260810 15:23:21.793592 21683 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
W20260810 15:23:21.793772 21685 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:21.793845 21685 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e822f28d3a7741a3b8eb56e3aee2c460; no voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57
I20260810 15:23:21.793956 22138 raft_consensus.cc:2726] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:23:21.839454 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:21.839198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:38599)
0810 15:23:21.839355 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:21.839408 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260810 15:23:22.667193 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:22.666938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:35443)
0810 15:23:22.667091 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:22.667142 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260810 15:23:22.704178 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:22.703948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:37791)
0810 15:23:22.704095 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:22.704132 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260810 15:23:22.840297 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:22.840046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:52581)
0810 15:23:22.840201 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:22.840247 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260810 15:23:23.497872 22147 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:23.497978 22147 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:23.498142 22147 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285), 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
W20260810 15:23:23.498590 21685 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
W20260810 15:23:23.498680 21683 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:23.498757 21683 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e822f28d3a7741a3b8eb56e3aee2c460; no voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57
I20260810 15:23:23.498855 22147 raft_consensus.cc:2726] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:23:23.668092 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:23.667830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:57273)
0810 15:23:23.668000 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:23.668041 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260810 15:23:23.705029 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:23.704723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:48933)
0810 15:23:23.704896 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:23.704963 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260810 15:23:23.841137 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:23.840895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:41481)
0810 15:23:23.841041 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:23.841088 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260810 15:23:24.668967 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:24.668736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:54909)
0810 15:23:24.668875 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:24.668917 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260810 15:23:24.706049 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:24.705787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:42763)
0810 15:23:24.705950 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:24.705995 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260810 15:23:24.842084 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:24.841816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:53879)
0810 15:23:24.841988 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:24.842033 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260810 15:23:25.669866 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:25.669576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:55163)
0810 15:23:25.669767 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:25.669817 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":132,"threads_started":1}
I20260810 15:23:25.706892 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:25.706594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:36853)
0810 15:23:25.706792 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:25.706838 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":111,"threads_started":1}
I20260810 15:23:25.842877 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:25.842591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:45607)
0810 15:23:25.842777 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:25.842826 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260810 15:23:26.062583 22161 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:26.062707 22161 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:26.062994 22161 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285), 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
W20260810 15:23:26.063447 21683 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
W20260810 15:23:26.063611 21685 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:26.063690 21685 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e822f28d3a7741a3b8eb56e3aee2c460; no voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57
I20260810 15:23:26.063809 22161 raft_consensus.cc:2726] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:23:26.670637 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:26.670382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:48465)
0810 15:23:26.670538 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:26.670583 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260810 15:23:26.707741 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:26.707520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:44685)
0810 15:23:26.707650 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:26.707692 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260810 15:23:26.843732 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:26.843471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:51901)
0810 15:23:26.843634 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:26.843679 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260810 15:23:27.671454 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:27.671227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:52421)
0810 15:23:27.671366 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:27.671407 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260810 15:23:27.708441 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:27.708224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:33393)
0810 15:23:27.708357 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:27.708392 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260810 15:23:27.844622 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:27.844344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:43857)
0810 15:23:27.844521 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:27.844573 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260810 15:23:28.672305 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:28.672021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:33241)
0810 15:23:28.672168 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:28.672213 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260810 15:23:28.709184 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:28.708947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:55301)
0810 15:23:28.709094 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:28.709135 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260810 15:23:28.838095 22171 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:28.838271 22171 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:28.838452 22171 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285), 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
W20260810 15:23:28.838887 21683 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
W20260810 15:23:28.838963 21685 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:28.839072 21685 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e822f28d3a7741a3b8eb56e3aee2c460; no voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57
I20260810 15:23:28.839147 22171 raft_consensus.cc:2726] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:23:28.845395 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:28.845087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:53687)
0810 15:23:28.845246 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:28.845286 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260810 15:23:29.673137 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:29.672872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:48979)
0810 15:23:29.673045 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:29.673088 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260810 15:23:29.709993 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:29.709673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:38825)
0810 15:23:29.709887 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:29.709944 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":121,"threads_started":1}
I20260810 15:23:29.846150 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:29.845919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:54265)
0810 15:23:29.846057 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:29.846103 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260810 15:23:30.674029 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:30.673776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:60735)
0810 15:23:30.673935 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:30.673979 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260810 15:23:30.710734 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:30.710480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:45697)
0810 15:23:30.710636 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:30.710683 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260810 15:23:30.846997 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:30.846753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:47351)
0810 15:23:30.846899 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:30.846945 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260810 15:23:31.675037 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:31.674754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:57115)
0810 15:23:31.674939 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:31.674986 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260810 15:23:31.711477 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:31.711220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:46837)
0810 15:23:31.711389 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:31.711426 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260810 15:23:31.847746 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:31.847484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:56313)
0810 15:23:31.847655 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:31.847694 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260810 15:23:32.676030 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:32.675717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:48395)
0810 15:23:32.675933 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:32.675980 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260810 15:23:32.712221 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:32.711985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:51759)
0810 15:23:32.712127 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:32.712171 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260810 15:23:32.848569 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:32.848319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:50297)
0810 15:23:32.848476 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:32.848519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260810 15:23:33.676858 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:33.676585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:58799)
0810 15:23:33.676759 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:33.676805 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260810 15:23:33.712981 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:33.712756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:47277)
0810 15:23:33.712891 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:33.712931 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260810 15:23:33.849390 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:33.849138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:34499)
0810 15:23:33.849285 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:33.849331 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260810 15:23:34.677796 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:34.677489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:52693)
0810 15:23:34.677698 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:34.677747 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260810 15:23:34.679191 22193 raft_consensus.cc:482] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:34.679281 22193 raft_consensus.cc:504] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e555807f830a490a9a571637e90b9f57" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 44285 } } peers { permanent_uuid: "410d8070b5c8497da243188b8add0f6b" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38903 } } peers { permanent_uuid: "e822f28d3a7741a3b8eb56e3aee2c460" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 34585 } }
I20260810 15:23:34.679489 22193 leader_election.cc:288] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285), 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903)
W20260810 15:23:34.679931 21683 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e555807f830a490a9a571637e90b9f57 (127.4.64.67:44285): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
W20260810 15:23:34.680011 21685 leader_election.cc:341] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 410d8070b5c8497da243188b8add0f6b (127.4.64.66:38903): Not found: Tablet not found: 4ebecb8d7b9d4fea98a45a72dfe31f2f
I20260810 15:23:34.680075 21685 leader_election.cc:302] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e822f28d3a7741a3b8eb56e3aee2c460; no voters: 410d8070b5c8497da243188b8add0f6b, e555807f830a490a9a571637e90b9f57
I20260810 15:23:34.680163 22193 raft_consensus.cc:2726] T 4ebecb8d7b9d4fea98a45a72dfe31f2f P e822f28d3a7741a3b8eb56e3aee2c460 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:23:34.713845 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:34.713530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:38253)
0810 15:23:34.713753 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:34.713793 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":137,"threads_started":1}
I20260810 15:23:34.850214 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:34.849976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:50237)
0810 15:23:34.850125 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:34.850164 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260810 15:23:35.678665 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:35.678406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.65:46809)
0810 15:23:35.678569 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:35.678615 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260810 15:23:35.714701 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:35.714434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.66:41339)
0810 15:23:35.714598 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:35.714652 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260810 15:23:35.843425 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 21632
I20260810 15:23:35.851013 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:35.850786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:44711 (local address 127.4.64.67:59947)
0810 15:23:35.850927 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:35.850964 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:44711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260810 15:23:35.853354 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 21823
I20260810 15:23:35.861740 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 21979
2026-08-10T15:23:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19459 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:23:35 dist-test-slave-9zdj krb5kdc[22203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:23:35 dist-test-slave-9zdj krb5kdc[22203](info): set up 2 sockets
Aug 10 15:23:35 dist-test-slave-9zdj krb5kdc[22203](info): commencing operation
krb5kdc: starting...
W20260810 15:23:37.029340 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:23:37.172761 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34487
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:37.258406 22219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:37.258658 22219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:37.258698 22219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:37.259886 22219 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:23:37.259954 22219 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:23:37.259976 22219 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:23:37.259996 22219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:37.260023 22219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:37.260056 22219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:37.260082 22219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:37.260114 22219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:37.261689 22219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:37.261996 22219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:37.262255 22219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:37.269134 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.269146 22225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:37.269146 22219 server_base.cc:1034] running on GCE node
W20260810 15:23:37.269222 22224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:37.269596 22219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:37.269951 22219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:37.271097 22219 hybrid_clock.cc:648] HybridClock initialized: now 1786375417271082 us; error 33 us; skew 500 ppm
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:37.273871 22219 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:37.274410 22219 webserver.cc:460] Webserver started at http://127.4.64.126:41825/ using document root <none> and password file <none>
I20260810 15:23:37.274685 22219 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:37.274760 22219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:37.274925 22219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:37.276108 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "cd7559e88849428295cd69973ee5f7de"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "523c45a7267d03c5dff1b72d69a4dfc2"
server_key_iv: "ab9904f3467ae65255392efb4ca59219"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.276540 22219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "cd7559e88849428295cd69973ee5f7de"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "523c45a7267d03c5dff1b72d69a4dfc2"
server_key_iv: "ab9904f3467ae65255392efb4ca59219"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.278072 22219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:23:37.278882 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.279104 22219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:37.279218 22219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "cd7559e88849428295cd69973ee5f7de"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "523c45a7267d03c5dff1b72d69a4dfc2"
server_key_iv: "ab9904f3467ae65255392efb4ca59219"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.279301 22219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:37.288800 22219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:37.290940 22219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:37.291123 22219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:37.295423 22219 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34487
I20260810 15:23:37.295476 22298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34487 every 8 connection(s)
I20260810 15:23:37.295893 22219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:37.296784 22299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:37.298879 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 22219
I20260810 15:23:37.298975 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:23:37.299202 4353 external_mini_cluster.cc:1428] Setting key 78166f8d0c5729eff5db9d07438ef5e8
I20260810 15:23:37.299521 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap starting.
I20260810 15:23:37.300231 22299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:37.300549 22299 log.cc:824] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:37.301400 22299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: No bootstrap required, opened a new log
I20260810 15:23:37.303661 22299 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:37.303815 22299 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:37.303845 22299 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7559e88849428295cd69973ee5f7de, State: Initialized, Role: FOLLOWER
I20260810 15:23:37.304001 22299 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:37.304075 22299 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:37.304097 22299 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:37.304165 22299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:37.304977 22299 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:37.305102 22299 leader_election.cc:302] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7559e88849428295cd69973ee5f7de; no voters:
I20260810 15:23:37.305285 22299 leader_election.cc:288] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:23:37.305371 22304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:37.305604 22299 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:37.305605 22304 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 LEADER]: Becoming Leader. State: Replica: cd7559e88849428295cd69973ee5f7de, State: Running, Role: LEADER
I20260810 15:23:37.305835 22304 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:37.306275 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7559e88849428295cd69973ee5f7de. Latest consensus state: current_term: 1 leader_uuid: "cd7559e88849428295cd69973ee5f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } } }
I20260810 15:23:37.306439 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: This master's current role is: LEADER
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:37.307090 22313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:37.307324 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7559e88849428295cd69973ee5f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } } }
I20260810 15:23:37.307550 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:37.307828 22313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:37.309748 22313 catalog_manager.cc:1347] Generated new cluster ID: 85c8f11b136444f2bc4a39d98d41e5c5
I20260810 15:23:37.309810 22313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:23:37.311909 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.299722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.4.64.126:34487)
0810 15:23:37.299908 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.299919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.299960 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:23:37.300465 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.300472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.300522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.300708 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:37.301523 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.302486 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.303624 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.304236 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.304346 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.307615 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:37.307619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:37.307620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:37.307633 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:37.310079 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.310507 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.310510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.310511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.310629 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.310862 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.310863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.310863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.311191 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:37.311343 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.311509 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.311672 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260810 15:23:37.319670 22313 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:23:37.320230 22313 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:37.328507 22313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Generated new TSK 0
I20260810 15:23:37.328744 22313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:23:37.361392 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34487
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:37.447445 22325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:37.447703 22325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:37.447743 22325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:37.448920 22325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:37.448993 22325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:37.449035 22325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:37.450522 22325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:37.450804 22325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:37.451051 22325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:37.457258 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.457244 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.457244 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:37.457412 22325 server_base.cc:1034] running on GCE node
I20260810 15:23:37.457779 22325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:37.458035 22325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:37.459249 22325 hybrid_clock.cc:648] HybridClock initialized: now 1786375417459240 us; error 48 us; skew 500 ppm
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:37.461896 22325 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:37.462443 22325 webserver.cc:460] Webserver started at http://127.4.64.65:43903/ using document root <none> and password file <none>
I20260810 15:23:37.462682 22325 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:37.462751 22325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:37.462908 22325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:37.463982 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "d1363a9e45e746b9884ca5c089a93802"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "0f7234b5fb872395c0e14d2f2dad3bb3"
server_key_iv: "580e4c9ad6312608ba912ddbf7222d3c"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.464361 22325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "d1363a9e45e746b9884ca5c089a93802"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "0f7234b5fb872395c0e14d2f2dad3bb3"
server_key_iv: "580e4c9ad6312608ba912ddbf7222d3c"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.465790 22325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:23:37.466604 22344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.466792 22325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:23:37.466892 22325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "d1363a9e45e746b9884ca5c089a93802"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "0f7234b5fb872395c0e14d2f2dad3bb3"
server_key_iv: "580e4c9ad6312608ba912ddbf7222d3c"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.466979 22325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:37.476840 22325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:37.479282 22325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:37.479452 22325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:37.479691 22325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:37.480003 22325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:37.480057 22325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.480093 22325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:37.480120 22325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.486558 22325 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:44931
I20260810 15:23:37.486598 22474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:44931 every 8 connection(s)
I20260810 15:23:37.486944 22325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:37.488152 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 22325
I20260810 15:23:37.488270 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:23:37.488536 4353 external_mini_cluster.cc:1428] Setting key 25581e9fd1ad09bfeacb670507871199
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:37.497339 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.487676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:43313 (local address 127.4.64.126:34487)
0810 15:23:37.487747 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.487749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.487953 (+ 204us) server_negotiation.cc:407] Connection header received
0810 15:23:37.488546 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.488547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.488561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.488646 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:37.489864 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.490629 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.491628 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.491974 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.492054 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.494422 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:37.494423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:37.494424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:37.494432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:37.495885 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.496328 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.496329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.496329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.496393 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.496662 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.496663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.496664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.496786 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:37.496861 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.497148 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.497243 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:23:37.497541 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.487632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:43313)
0810 15:23:37.487820 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.487864 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.488438 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.488744 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.488756 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.489488 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:37.489764 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.489769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.490732 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.490733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.491542 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.491544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.492065 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.492066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.492174 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.492728 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:37.492737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:37.494321 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:37.495995 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:37.496002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:37.496005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:37.496238 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:37.496467 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:37.496468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:37.496469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:37.496580 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:37.496865 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:37.496872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:37.497017 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.497128 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.497298 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
I20260810 15:23:37.498075 22475 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34487
I20260810 15:23:37.498234 22475 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:37.498448 22475 heartbeater.cc:511] Master 127.4.64.126:34487 requested a full tablet report, sending...
I20260810 15:23:37.499037 22258 ts_manager.cc:194] Registered new tserver with Master: d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931)
I20260810 15:23:37.499665 22258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:43313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:23:37.518399 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34487
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:37.608497 22481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:37.608758 22481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:37.608808 22481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:37.610018 22481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:37.610090 22481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:37.610134 22481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:37.611631 22481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:37.611955 22481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:37.612243 22481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:37.617635 22492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.617682 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.617657 22490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:37.617684 22481 server_base.cc:1034] running on GCE node
I20260810 15:23:37.618076 22481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:37.618330 22481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:37.619467 22481 hybrid_clock.cc:648] HybridClock initialized: now 1786375417619456 us; error 32 us; skew 500 ppm
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:37.622326 22481 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:37.622839 22481 webserver.cc:460] Webserver started at http://127.4.64.66:35065/ using document root <none> and password file <none>
I20260810 15:23:37.623092 22481 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:37.623163 22481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:37.623315 22481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:37.624382 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "4818ba371a1b40e296a97d2fa5ca33d0"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "5aa493b0da0e871513fc0fe9b2b66750"
server_key_iv: "f8b41b9959fefc04182fd4c9f4c31dc3"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.624749 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "4818ba371a1b40e296a97d2fa5ca33d0"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "5aa493b0da0e871513fc0fe9b2b66750"
server_key_iv: "f8b41b9959fefc04182fd4c9f4c31dc3"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.626242 22481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:23:37.627066 22500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.627275 22481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:37.627388 22481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "4818ba371a1b40e296a97d2fa5ca33d0"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "5aa493b0da0e871513fc0fe9b2b66750"
server_key_iv: "f8b41b9959fefc04182fd4c9f4c31dc3"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.627463 22481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:37.644584 22481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:37.647276 22481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:37.647464 22481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:37.647735 22481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:37.648116 22481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:37.648180 22481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.648228 22481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:37.648253 22481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.654356 22481 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:45525
I20260810 15:23:37.654408 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:45525 every 8 connection(s)
I20260810 15:23:37.654803 22481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:37.664096 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.655618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:53169 (local address 127.4.64.126:34487)
0810 15:23:37.655662 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.655664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.655895 (+ 231us) server_negotiation.cc:407] Connection header received
0810 15:23:37.656369 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.656370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.656381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.656437 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:37.657408 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.657939 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.658931 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.659120 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.659175 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.661407 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:37.661408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:37.661409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:37.661415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:37.662755 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.663157 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.663158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.663159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.663210 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.663447 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.663447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.663447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.663552 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:37.663603 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.663938 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.664027 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:23:37.664252 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.655605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:53169)
0810 15:23:37.655761 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.655804 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.656274 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.656499 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.656503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.657114 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:37.657326 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.657332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.658024 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.658025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.658843 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.658845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.659199 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.659199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.659312 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.659880 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:37.659889 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:37.661312 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:37.662846 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:37.662856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:37.662858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:37.663084 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:37.663267 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:37.663268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:37.663268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:37.663388 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:37.663630 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:37.663637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:37.663765 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.663869 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.664000 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260810 15:23:37.664776 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 22481
I20260810 15:23:37.664880 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:23:37.664841 22629 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34487
I20260810 15:23:37.664958 22629 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:37.665143 4353 external_mini_cluster.cc:1428] Setting key 708eb99af024ad3f39d625c3989c4d7a
I20260810 15:23:37.665181 22629 heartbeater.cc:511] Master 127.4.64.126:34487 requested a full tablet report, sending...
I20260810 15:23:37.665639 22257 ts_manager.cc:194] Registered new tserver with Master: 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
I20260810 15:23:37.666085 22257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:53169
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:23:37.696413 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34487
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:37.780889 22635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:37.781133 22635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:37.781172 22635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:37.782416 22635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:37.782485 22635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:37.782529 22635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:37.784058 22635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:37.784344 22635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:37.784595 22635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:37.790203 22643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.790244 22642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:37.790203 22645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:37.790585 22635 server_base.cc:1034] running on GCE node
I20260810 15:23:37.790840 22635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:37.791092 22635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:37.792241 22635 hybrid_clock.cc:648] HybridClock initialized: now 1786375417792224 us; error 28 us; skew 500 ppm
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:37.795137 22635 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:37.795589 22635 webserver.cc:460] Webserver started at http://127.4.64.67:39233/ using document root <none> and password file <none>
I20260810 15:23:37.795846 22635 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:37.795917 22635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:37.796064 22635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:23:37.797173 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "df61a2c31df7499d8d294330d2f358af"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "22eee2fd8345a9088cb136319ecf08cb"
server_key_iv: "350c913090ed5b6d331baf2d29b32003"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.797546 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "df61a2c31df7499d8d294330d2f358af"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "22eee2fd8345a9088cb136319ecf08cb"
server_key_iv: "350c913090ed5b6d331baf2d29b32003"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.799063 22635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:37.799816 22654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.800004 22635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.800112 22635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "df61a2c31df7499d8d294330d2f358af"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "22eee2fd8345a9088cb136319ecf08cb"
server_key_iv: "350c913090ed5b6d331baf2d29b32003"
server_key_version: "encryptionkey@0"
I20260810 15:23:37.800187 22635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:37.826391 22635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:37.828989 22635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:37.829175 22635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:37.829444 22635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:37.829851 22635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:37.829907 22635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.829957 22635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:37.830021 22635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:37.836465 22635 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:40813
I20260810 15:23:37.836514 22792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:40813 every 8 connection(s)
I20260810 15:23:37.836905 22635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:23:37.842694 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 22635
I20260810 15:23:37.842824 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:23:37.843043 4353 external_mini_cluster.cc:1428] Setting key 08c4c8d7a96f8322a69b1c1bb4e522e1
Aug 10 15:23:37 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:37.847021 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.837682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:60665 (local address 127.4.64.126:34487)
0810 15:23:37.837727 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.837729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.838002 (+ 273us) server_negotiation.cc:407] Connection header received
0810 15:23:37.838570 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.838572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.838587 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.838639 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:37.839699 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.840222 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.841285 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.841496 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.841576 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.844390 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:37.844391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:37.844391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:37.844399 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:37.845783 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.846172 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.846173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.846173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.846226 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.846477 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.846477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.846478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.846586 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:37.846638 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.846890 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.846954 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:23:37.847223 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.837692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:60665)
0810 15:23:37.837869 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.837909 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.838449 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.838714 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.838718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.839358 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:37.839578 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.839583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.840324 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.840325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.841189 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.841191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.841586 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.841587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.841749 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.842325 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:37.842335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:37.844271 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:37.845858 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:37.845864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:37.845866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:37.846101 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:37.846284 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:37.846311 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:37.846311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:37.846421 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:37.846640 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:37.846645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:37.846772 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.846859 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.846990 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260810 15:23:37.847733 22793 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34487
I20260810 15:23:37.847862 22793 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:37.848078 22793 heartbeater.cc:511] Master 127.4.64.126:34487 requested a full tablet report, sending...
I20260810 15:23:37.848475 22257 ts_manager.cc:194] Registered new tserver with Master: df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813)
I20260810 15:23:37.848876 22257 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:60665
I20260810 15:23:37.855120 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:23:37.863171 22302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.856470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.4.64.126:34487)
0810 15:23:37.856537 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.856539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.856697 (+ 158us) server_negotiation.cc:407] Connection header received
0810 15:23:37.856751 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.856752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.856762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.856812 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:37.857509 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.858066 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.858935 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.859122 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.859198 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.860617 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:37.860618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:37.860618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:37.860625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:37.861997 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.862322 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.862323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.862323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.862394 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:37.862561 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:37.862562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:37.862562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:37.862705 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:37.862780 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.862918 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.863038 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:23:37.865028 22257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:23:37.866791 22257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:23:37.873962 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.869786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.4.64.66:45525)
0810 15:23:37.869963 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.869967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.870349 (+ 382us) server_negotiation.cc:407] Connection header received
0810 15:23:37.870423 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.870425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.870441 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.870521 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:37.870737 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.871786 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.872957 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.873481 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873575 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.873662 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.873825 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.873869 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260810 15:23:37.873950 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.869759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:45525 (local address 127.0.0.1:52508)
0810 15:23:37.870285 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.870296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.870352 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.870610 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.870611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.870614 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:37.870671 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.870673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.871892 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.871893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.872863 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.872866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.873574 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.873681 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.873697 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.873805 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.873870 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":72,"threads_started":1}
I20260810 15:23:37.874183 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.869441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:40813 (local address 127.0.0.1:59056)
0810 15:23:37.869884 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.869895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.869950 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.870552 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.870553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.870555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:37.870619 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.870621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.871499 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.871500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.872914 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.872918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.873924 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.874004 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.874009 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.874040 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.874091 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":53,"thread_start_us":100,"threads_started":1}
I20260810 15:23:37.874368 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.869507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.4.64.65:44931)
0810 15:23:37.869677 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.869682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.869824 (+ 142us) server_negotiation.cc:407] Connection header received
0810 15:23:37.869993 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.869996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.870019 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.870109 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:37.870403 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.871595 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873024 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.873832 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873892 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.873948 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.874237 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.874282 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260810 15:23:37.874197 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.869382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44931 (local address 127.0.0.1:49880)
0810 15:23:37.869705 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.869750 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.869854 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.870134 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.870138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.870141 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:37.870255 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.870257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.871708 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.871709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.872916 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.872918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.873977 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.874032 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.874036 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.874083 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.874113 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260810 15:23:37.874547 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.869562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.4.64.67:40813)
0810 15:23:37.870072 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.870074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.870082 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:23:37.870425 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.870428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.870445 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.870549 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:37.870692 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.871388 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.873024 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.873816 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.874324 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.874392 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.874418 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.874463 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":133,"threads_started":1}
I20260810 15:23:37.875290 22559 tablet_service.cc:1467] Processing CreateTablet for tablet 8b07831bf61945abbe0b08444779425f (DEFAULT_TABLE table=test-table [id=c106b2483225444e87363d192cbff0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:37.875291 22715 tablet_service.cc:1467] Processing CreateTablet for tablet 8b07831bf61945abbe0b08444779425f (DEFAULT_TABLE table=test-table [id=c106b2483225444e87363d192cbff0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:37.875689 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b07831bf61945abbe0b08444779425f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:37.875687 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b07831bf61945abbe0b08444779425f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:37.878033 22400 tablet_service.cc:1467] Processing CreateTablet for tablet 8b07831bf61945abbe0b08444779425f (DEFAULT_TABLE table=test-table [id=c106b2483225444e87363d192cbff0e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:37.878386 22400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b07831bf61945abbe0b08444779425f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:37.879258 22815 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Bootstrap starting.
I20260810 15:23:37.879985 22815 tablet_bootstrap.cc:654] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:37.880636 22815 log.cc:824] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:37.881036 22814 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Bootstrap starting.
I20260810 15:23:37.881474 22815 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: No bootstrap required, opened a new log
I20260810 15:23:37.881572 22815 ts_tablet_manager.cc:1397] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:37.881772 22814 tablet_bootstrap.cc:654] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:37.881843 22817 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Bootstrap starting.
I20260810 15:23:37.882460 22814 log.cc:824] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:37.882619 22817 tablet_bootstrap.cc:654] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:37.883242 22817 log.cc:824] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:37.883272 22814 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: No bootstrap required, opened a new log
I20260810 15:23:37.883337 22814 ts_tablet_manager.cc:1397] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:23:37.884127 22817 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: No bootstrap required, opened a new log
I20260810 15:23:37.883989 22815 raft_consensus.cc:348] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.884176 22815 raft_consensus.cc:374] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:37.884199 22817 ts_tablet_manager.cc:1397] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:37.884214 22815 raft_consensus.cc:729] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df61a2c31df7499d8d294330d2f358af, State: Initialized, Role: FOLLOWER
I20260810 15:23:37.884377 22815 consensus_queue.cc:260] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.884678 22815 ts_tablet_manager.cc:1428] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:23:37.884846 22793 heartbeater.cc:503] Master 127.4.64.126:34487 was elected leader, sending a full tablet report...
I20260810 15:23:37.885350 22814 raft_consensus.cc:348] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.885572 22814 raft_consensus.cc:374] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:37.885643 22814 raft_consensus.cc:729] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4818ba371a1b40e296a97d2fa5ca33d0, State: Initialized, Role: FOLLOWER
I20260810 15:23:37.885767 22814 consensus_queue.cc:260] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.886057 22814 ts_tablet_manager.cc:1428] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:23:37.886184 22629 heartbeater.cc:503] Master 127.4.64.126:34487 was elected leader, sending a full tablet report...
I20260810 15:23:37.886235 22817 raft_consensus.cc:348] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.886466 22817 raft_consensus.cc:374] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:37.886515 22817 raft_consensus.cc:729] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1363a9e45e746b9884ca5c089a93802, State: Initialized, Role: FOLLOWER
I20260810 15:23:37.886660 22817 consensus_queue.cc:260] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.887019 22475 heartbeater.cc:503] Master 127.4.64.126:34487 was elected leader, sending a full tablet report...
I20260810 15:23:37.887002 22817 ts_tablet_manager.cc:1428] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260810 15:23:37.905432 22631 tablet.cc:2369] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:37.968339 22822 raft_consensus.cc:482] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:37.968474 22822 raft_consensus.cc:504] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.968863 22822 leader_election.cc:288] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931), df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813)
I20260810 15:23:37.971853 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.969034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44931 (local address 127.4.64.66:52753)
0810 15:23:37.969095 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.969107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.969151 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.969298 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.969299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.969301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:37.969382 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.969384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.970108 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.970109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.970978 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.970981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.971567 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.971568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.971660 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.971663 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.971733 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.971777 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:23:37.971853 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.969113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:52753 (local address 127.4.64.65:44931)
0810 15:23:37.969157 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.969159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.969191 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:23:37.969222 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.969223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.969236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.969284 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:37.969461 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.970018 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.971082 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.971478 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.971566 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.971577 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.971739 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.971780 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:23:37.972050 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.969158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:36129 (local address 127.4.64.67:40813)
0810 15:23:37.969225 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.969227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.969452 (+ 225us) server_negotiation.cc:407] Connection header received
0810 15:23:37.969524 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.969525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.969537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.969612 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:37.969805 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.970307 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.971307 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.971694 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.971776 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.971787 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.971946 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.971984 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 15:23:37.972062 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.969071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:40813 (local address 127.4.64.66:36129)
0810 15:23:37.969295 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:37.969388 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:37.969466 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:37.969640 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:37.969641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:37.969643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:37.969724 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.969726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.970398 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.970399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.971218 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:37.971220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.971793 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.971794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:37.971873 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.971876 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:23:37.971943 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:23:37.971982 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":138,"threads_started":1}
I20260810 15:23:37.972244 22422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07831bf61945abbe0b08444779425f" candidate_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1363a9e45e746b9884ca5c089a93802" is_pre_election: true
I20260810 15:23:37.972425 22422 raft_consensus.cc:2454] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4818ba371a1b40e296a97d2fa5ca33d0 in term 0.
I20260810 15:23:37.972386 22740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07831bf61945abbe0b08444779425f" candidate_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df61a2c31df7499d8d294330d2f358af" is_pre_election: true
I20260810 15:23:37.972532 22740 raft_consensus.cc:2454] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4818ba371a1b40e296a97d2fa5ca33d0 in term 0.
I20260810 15:23:37.972673 22502 leader_election.cc:302] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4818ba371a1b40e296a97d2fa5ca33d0, d1363a9e45e746b9884ca5c089a93802; no voters:
I20260810 15:23:37.972877 22822 raft_consensus.cc:2781] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:23:37.972937 22822 raft_consensus.cc:482] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:23:37.972966 22822 raft_consensus.cc:3037] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:37.973733 22822 raft_consensus.cc:504] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.973897 22822 leader_election.cc:288] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [CANDIDATE]: Term 1 election: Requested vote from peers d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931), df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813)
I20260810 15:23:37.974066 22740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07831bf61945abbe0b08444779425f" candidate_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df61a2c31df7499d8d294330d2f358af"
I20260810 15:23:37.974083 22422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07831bf61945abbe0b08444779425f" candidate_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1363a9e45e746b9884ca5c089a93802"
I20260810 15:23:37.974144 22422 raft_consensus.cc:3037] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:37.974165 22740 raft_consensus.cc:3037] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:37.974859 22422 raft_consensus.cc:2454] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4818ba371a1b40e296a97d2fa5ca33d0 in term 1.
I20260810 15:23:37.974980 22740 raft_consensus.cc:2454] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4818ba371a1b40e296a97d2fa5ca33d0 in term 1.
I20260810 15:23:37.975030 22502 leader_election.cc:302] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4818ba371a1b40e296a97d2fa5ca33d0, d1363a9e45e746b9884ca5c089a93802; no voters:
I20260810 15:23:37.975124 22822 raft_consensus.cc:2781] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:37.975306 22822 raft_consensus.cc:686] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 LEADER]: Becoming Leader. State: Replica: 4818ba371a1b40e296a97d2fa5ca33d0, State: Running, Role: LEADER
I20260810 15:23:37.975414 22822 consensus_queue.cc:237] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:37.976439 22258 catalog_manager.cc:5495] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66). New cstate: current_term: 1 leader_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } health_report { overall_health: UNKNOWN } } }
W20260810 15:23:37.987690 22476 tablet.cc:2369] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:37.994527 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:37.992245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.4.64.66:45525)
0810 15:23:37.992303 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:37.992307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:37.992321 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:23:37.992381 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:37.992382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:37.992401 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:37.992463 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:37.992581 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.993057 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.993853 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:37.994038 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:37.994091 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:37.994215 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:37.994313 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:23:37.994379 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:37.994413 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 15:23:37.994431 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:23:37.996340 22422 raft_consensus.cc:1261] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Refusing update from remote peer 4818ba371a1b40e296a97d2fa5ca33d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:37.996347 22740 raft_consensus.cc:1261] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Refusing update from remote peer 4818ba371a1b40e296a97d2fa5ca33d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:37.996707 22822 consensus_queue.cc:1035] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:37.996855 22825 consensus_queue.cc:1035] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:38.001984 22257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:38.002074 22257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:38.002889 22257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Sending DeleteTablet for 3 replicas of tablet 8b07831bf61945abbe0b08444779425f
I20260810 15:23:38.003225 22715 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07831bf61945abbe0b08444779425f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:59056
I20260810 15:23:38.003302 22559 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07831bf61945abbe0b08444779425f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:52508
I20260810 15:23:38.003319 22400 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07831bf61945abbe0b08444779425f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:49880
I20260810 15:23:38.003542 22840 tablet_replica.cc:330] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: stopping tablet replica
I20260810 15:23:38.003557 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 22325
I20260810 15:23:38.003583 22838 tablet_replica.cc:330] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: stopping tablet replica
I20260810 15:23:38.003722 22838 raft_consensus.cc:2229] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:38.003723 22840 raft_consensus.cc:2229] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:38.003839 22838 raft_consensus.cc:2258] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:38.004011 22839 tablet_replica.cc:330] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: stopping tablet replica
I20260810 15:23:38.005168 22838 ts_tablet_manager.cc:1905] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:38.004113 22839 raft_consensus.cc:2229] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:23:38.006897 22839 raft_consensus.cc:2258] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:38.007179 22838 ts_tablet_manager.cc:1918] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:38.007253 22838 log.cc:1196] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/8b07831bf61945abbe0b08444779425f
I20260810 15:23:38.007537 22838 ts_tablet_manager.cc:1939] T 8b07831bf61945abbe0b08444779425f P df61a2c31df7499d8d294330d2f358af: Deleting consensus metadata
I20260810 15:23:38.007985 22239 catalog_manager.cc:4841] TS df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): tablet 8b07831bf61945abbe0b08444779425f (table test-table [id=c106b2483225444e87363d192cbff0e6]) successfully deleted
I20260810 15:23:38.008101 22839 ts_tablet_manager.cc:1905] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:38.009511 22839 ts_tablet_manager.cc:1918] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:38.009599 22839 log.cc:1196] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/8b07831bf61945abbe0b08444779425f
I20260810 15:23:38.009930 22839 ts_tablet_manager.cc:1939] T 8b07831bf61945abbe0b08444779425f P 4818ba371a1b40e296a97d2fa5ca33d0: Deleting consensus metadata
I20260810 15:23:38.010380 22240 catalog_manager.cc:4841] TS 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): tablet 8b07831bf61945abbe0b08444779425f (table test-table [id=c106b2483225444e87363d192cbff0e6]) successfully deleted
W20260810 15:23:38.016973 22240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:44931 (error 108)
I20260810 15:23:38.017130 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 22481
I20260810 15:23:38.018558 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:38.018435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44931 (local address 127.0.0.1:49894)
0810 15:23:38.018483 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:38.018514 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:44931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 15:23:38.018680 22240 catalog_manager.cc:4584] TS d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b07831bf61945abbe0b08444779425f: Network error: Client connection negotiation failed: client connection to 127.4.64.65:44931: connect: Connection refused (error 111)
I20260810 15:23:38.025199 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 22635
I20260810 15:23:38.032464 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 22219
I20260810 15:23:58.040242 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34487
--webserver_interface=127.4.64.126
--webserver_port=41825
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:23:58.136054 22843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:58.136338 22843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:58.136379 22843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:58.137574 22843 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:23:58.137688 22843 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:23:58.137720 22843 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:23:58.137738 22843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:23:58.137763 22843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:58.137790 22843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:58.137825 22843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:23:58.137853 22843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:23:58.139424 22843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:58.139743 22843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:58.139999 22843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:58.150249 22852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:58.150327 22853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:58.150619 22843 server_base.cc:1034] running on GCE node
W20260810 15:23:58.150249 22855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:58.151021 22843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:58.151353 22843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:58.152503 22843 hybrid_clock.cc:648] HybridClock initialized: now 1786375438152483 us; error 40 us; skew 500 ppm
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:58.155395 22843 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:23:58.155916 22843 webserver.cc:460] Webserver started at http://127.4.64.126:41825/ using document root <none> and password file <none>
I20260810 15:23:58.156227 22843 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:58.156291 22843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:58.157779 22843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:23:58.158505 22862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.158758 22843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:58.158872 22843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "cd7559e88849428295cd69973ee5f7de"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "523c45a7267d03c5dff1b72d69a4dfc2"
server_key_iv: "ab9904f3467ae65255392efb4ca59219"
server_key_version: "encryptionkey@0"
I20260810 15:23:58.159261 22843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:58.181460 22843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:58.183878 22843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:58.184011 22843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:58.189020 22843 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34487
I20260810 15:23:58.189061 22923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34487 every 8 connection(s)
I20260810 15:23:58.189528 22843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:23:58.191226 22924 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:23:58.191996 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap starting.
I20260810 15:23:58.197374 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 22843
I20260810 15:23:58.197731 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:44931
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:34487
--webserver_port=43903
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:58.201707 22924 log.cc:824] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:58.203827 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:58.204182 22924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap complete.
I20260810 15:23:58.207082 22924 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:58.207389 22924 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7559e88849428295cd69973ee5f7de, State: Initialized, Role: FOLLOWER
I20260810 15:23:58.207615 22924 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:58.207716 22924 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:23:58.207782 22924 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:23:58.207834 22924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:23:58.208976 22924 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:58.209115 22924 leader_election.cc:302] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7559e88849428295cd69973ee5f7de; no voters:
I20260810 15:23:58.209355 22924 leader_election.cc:288] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:23:58.209398 22929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:23:58.209524 22929 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 LEADER]: Becoming Leader. State: Replica: cd7559e88849428295cd69973ee5f7de, State: Running, Role: LEADER
I20260810 15:23:58.209697 22924 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:23:58.209717 22929 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:23:58.210173 22930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd7559e88849428295cd69973ee5f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } } }
I20260810 15:23:58.210264 22930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:58.210647 22935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:23:58.210695 22931 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7559e88849428295cd69973ee5f7de. Latest consensus state: current_term: 2 leader_uuid: "cd7559e88849428295cd69973ee5f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } } }
I20260810 15:23:58.210858 22931 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: This master's current role is: LEADER
I20260810 15:23:58.211589 22935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:23:58.211705 22935 catalog_manager.cc:1259] Loaded cluster ID: 85c8f11b136444f2bc4a39d98d41e5c5
I20260810 15:23:58.211833 22935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:23:58.213393 22935 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:23:58.213590 22935 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Loaded TSK: 0
I20260810 15:23:58.230232 22935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Generated new TSK 1
I20260810 15:23:58.230374 22935 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:23:58.292748 22926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:58.292985 22926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:58.293010 22926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:58.294246 22926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:58.294292 22926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:58.294327 22926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:23:58.295853 22926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:44931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=43903
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:58.296139 22926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:58.296409 22926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:58.302888 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:58.303151 22926 server_base.cc:1034] running on GCE node
W20260810 15:23:58.302888 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:58.302881 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:58.303618 22926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:58.303933 22926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:58.305135 22926 hybrid_clock.cc:648] HybridClock initialized: now 1786375438305094 us; error 46 us; skew 500 ppm
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:58.308504 22926 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:23:58.309062 22926 webserver.cc:460] Webserver started at http://127.4.64.65:43903/ using document root <none> and password file <none>
I20260810 15:23:58.309334 22926 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:58.309425 22926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:58.311146 22926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:58.312067 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.312263 22926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:58.312371 22926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "d1363a9e45e746b9884ca5c089a93802"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "0f7234b5fb872395c0e14d2f2dad3bb3"
server_key_iv: "580e4c9ad6312608ba912ddbf7222d3c"
server_key_version: "encryptionkey@0"
I20260810 15:23:58.312870 22926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:58.337544 22926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:58.340138 22926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:58.340306 22926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:58.340574 22926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:58.341082 22973 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:23:58.342811 22926 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:23:58.342880 22926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:23:58.342916 22926 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:23:58.343798 22926 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:23:58.343847 22926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:23:58.343950 22973 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Bootstrap starting.
I20260810 15:23:58.353538 22926 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:44931
I20260810 15:23:58.353706 23099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:44931 every 8 connection(s)
I20260810 15:23:58.354036 22926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:23:58.354983 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 22926
I20260810 15:23:58.355311 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:45525
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:34487
--webserver_port=35065
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:58.357460 22973 log.cc:824] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:58.361449 22973 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:23:58.361912 22973 tablet_bootstrap.cc:492] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Bootstrap complete.
I20260810 15:23:58.363693 22973 ts_tablet_manager.cc:1397] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.006s
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:58.365190 22973 raft_consensus.cc:348] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:58.365604 22973 raft_consensus.cc:729] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1363a9e45e746b9884ca5c089a93802, State: Initialized, Role: FOLLOWER
I20260810 15:23:58.365873 22973 consensus_queue.cc:260] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } }
I20260810 15:23:58.366228 22973 ts_tablet_manager.cc:1428] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:58.369343 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.354808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:56379)
0810 15:23:58.355379 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.355434 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.356363 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.357011 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.357016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.357463 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:58.357837 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.357844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.359460 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.359461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.360780 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.360784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.361869 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.361871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.362031 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.362748 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:58.362760 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:58.364857 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:58.367597 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.367602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.367605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.367850 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.368182 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.368183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.368183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.368285 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.368806 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:58.368813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:58.368940 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.369027 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.369168 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":101,"threads_started":1}
I20260810 15:23:58.369676 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.354931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:56379 (local address 127.4.64.126:34487)
0810 15:23:58.355236 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.355248 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.355560 (+ 312us) server_negotiation.cc:407] Connection header received
0810 15:23:58.356616 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.356623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.356686 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.356853 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:58.357990 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.359315 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.360889 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.361571 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.361737 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.365003 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:58.365009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:58.365009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:58.365025 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:58.367490 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.367950 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.367955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.367955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.368099 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.368363 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.368364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.368365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.368725 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:58.368827 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.369131 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.369381 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260810 15:23:58.370347 23100 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34487
I20260810 15:23:58.370469 23100 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:58.370690 23100 heartbeater.cc:511] Master 127.4.64.126:34487 requested a full tablet report, sending...
I20260810 15:23:58.371555 22880 ts_manager.cc:194] Registered new tserver with Master: d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931)
I20260810 15:23:58.372835 22880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:56379
I20260810 15:23:58.373904 23100 heartbeater.cc:503] Master 127.4.64.126:34487 was elected leader, sending a full tablet report...
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.65@KRBTEST.COM
I20260810 15:23:58.381533 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.372226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44931 (local address 127.0.0.1:54338)
0810 15:23:58.373158 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.373191 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.373270 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.373564 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.373571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.373981 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:58.374164 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.374167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.375056 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.375057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.375699 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.375701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.376200 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.376201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.376291 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.376845 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:58.376849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:58.378461 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:58.380271 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.380279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.380282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.380498 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.380730 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.380731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.380732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.380809 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.381108 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:58.381117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:58.381176 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.381285 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.381432 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":101,"threads_started":1}
I20260810 15:23:58.381534 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.372422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.4.64.65:44931)
0810 15:23:58.373401 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.373406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.373415 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:23:58.373449 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.373451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.373470 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.373535 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:58.374231 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.374966 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.375801 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.376103 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.376196 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.378567 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:58.378579 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:58.378579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:58.378607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:58.380172 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.380591 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.380595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.380595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.380659 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.380884 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.380884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.380885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.381034 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:58.381132 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.381334 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.381429 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":68,"threads_started":1}
I20260810 15:23:58.382038 23016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07831bf61945abbe0b08444779425f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:37 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:54338
I20260810 15:23:58.382611 23111 tablet_replica.cc:330] stopping tablet replica
I20260810 15:23:58.382740 23111 raft_consensus.cc:2229] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:58.382813 23111 raft_consensus.cc:2258] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:58.383145 23111 ts_tablet_manager.cc:1905] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:58.385878 23111 ts_tablet_manager.cc:1918] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:58.385969 23111 log.cc:1196] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal/wals/8b07831bf61945abbe0b08444779425f
I20260810 15:23:58.386219 23111 ts_tablet_manager.cc:1939] T 8b07831bf61945abbe0b08444779425f P d1363a9e45e746b9884ca5c089a93802: Deleting consensus metadata
I20260810 15:23:58.386832 22864 catalog_manager.cc:4841] TS d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931): tablet 8b07831bf61945abbe0b08444779425f (table test-table [id=c106b2483225444e87363d192cbff0e6]) successfully deleted
W20260810 15:23:58.462055 23105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:58.462344 23105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:58.462383 23105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:58.463792 23105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:58.463840 23105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:58.463893 23105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:23:58.465705 23105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:45525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=35065
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:58.465982 23105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:58.466279 23105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:58.473383 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:58.473392 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:58.473439 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:58.473634 23105 server_base.cc:1034] running on GCE node
I20260810 15:23:58.473867 23105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:58.474195 23105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:58.475380 23105 hybrid_clock.cc:648] HybridClock initialized: now 1786375438475338 us; error 36 us; skew 500 ppm
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:58.479048 23105 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:23:58.479688 23105 webserver.cc:460] Webserver started at http://127.4.64.66:35065/ using document root <none> and password file <none>
I20260810 15:23:58.480005 23105 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:58.480100 23105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:58.482125 23105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:23:58.482826 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.483037 23105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:58.483144 23105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "4818ba371a1b40e296a97d2fa5ca33d0"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "5aa493b0da0e871513fc0fe9b2b66750"
server_key_iv: "f8b41b9959fefc04182fd4c9f4c31dc3"
server_key_version: "encryptionkey@0"
I20260810 15:23:58.483548 23105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:58.503979 23105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:58.506250 23105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:58.506453 23105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:58.506711 23105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:58.507051 23105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:58.507109 23105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.507155 23105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:58.507180 23105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.513911 23105 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:45525
I20260810 15:23:58.513999 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:45525 every 8 connection(s)
I20260810 15:23:58.514415 23105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:58.524084 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 23105
I20260810 15:23:58.524128 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.515173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:40779 (local address 127.4.64.126:34487)
0810 15:23:58.515219 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.515221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.515438 (+ 217us) server_negotiation.cc:407] Connection header received
0810 15:23:58.515955 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.515956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.515969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.516038 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:58.516998 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.517566 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.518629 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.518835 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.518892 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.521109 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:58.521110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:58.521111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:58.521121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:58.522643 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.523077 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.523078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.523079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.523143 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.523411 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.523412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.523412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.523534 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:58.523609 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.523931 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.524029 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:23:58.524340 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.515133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:40779)
0810 15:23:58.515316 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.515361 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.515866 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.516096 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.516100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.516709 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:58.516914 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.516919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.517711 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.517712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.518545 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.518547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.518914 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.518915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.519019 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.519537 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:58.519545 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:58.521003 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:58.522747 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.522755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.522758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.522996 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.523213 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.523215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.523215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.523340 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.523611 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:58.523619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:58.523767 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.523905 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.524079 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260810 15:23:58.524413 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:40813
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:34487
--webserver_port=39233
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:23:58.524904 23254 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34487
I20260810 15:23:58.525022 23254 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:58.525249 23254 heartbeater.cc:511] Master 127.4.64.126:34487 requested a full tablet report, sending...
I20260810 15:23:58.525681 22880 ts_manager.cc:194] Registered new tserver with Master: 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
I20260810 15:23:58.526073 22880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:40779
W20260810 15:23:58.610867 23258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:23:58.611155 23258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:23:58.611197 23258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:23:58.612367 23258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:23:58.612437 23258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:23:58.612479 23258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:23:58.613998 23258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=39233
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:23:58.614297 23258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:23:58.614555 23258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:23:58.620306 23268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:58.620386 23264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:23:58.620568 23265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:23:58.620591 23258 server_base.cc:1034] running on GCE node
I20260810 15:23:58.620985 23258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:23:58.621243 23258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:23:58.622440 23258 hybrid_clock.cc:648] HybridClock initialized: now 1786375438622429 us; error 43 us; skew 500 ppm
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:23:58.624944 23258 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:23:58.625450 23258 webserver.cc:460] Webserver started at http://127.4.64.67:39233/ using document root <none> and password file <none>
I20260810 15:23:58.625772 23258 fs_manager.cc:359] Metadata directory not provided
I20260810 15:23:58.625857 23258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:23:58.627323 23258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:58.627933 23275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.628126 23258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:23:58.628201 23258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "df61a2c31df7499d8d294330d2f358af"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "22eee2fd8345a9088cb136319ecf08cb"
server_key_iv: "350c913090ed5b6d331baf2d29b32003"
server_key_version: "encryptionkey@0"
I20260810 15:23:58.628612 23258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:23:58.638732 23258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:23:58.640592 23258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:23:58.640774 23258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:23:58.640964 23258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:23:58.641286 23258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:23:58.641321 23258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.641345 23258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:23:58.641392 23258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:23:58.647460 23258 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:40813
I20260810 15:23:58.647504 23401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:40813 every 8 connection(s)
I20260810 15:23:58.647876 23258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:23:58.651015 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 23258
Aug 10 15:23:58 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375438, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:23:58.658968 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.651465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.4.64.126:34487)
0810 15:23:58.651584 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.651586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.651713 (+ 127us) server_negotiation.cc:407] Connection header received
0810 15:23:58.651790 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.651791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.651800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.651861 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:58.652628 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.653250 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.654332 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.654519 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.654572 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.656098 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:58.656099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:58.656099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:58.656105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:58.657703 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.658080 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.658081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.658081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.658145 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.658360 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.658361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.658361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.658530 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:58.658704 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.658827 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.658897 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260810 15:23:58.659488 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.648598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:40051)
0810 15:23:58.648787 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.648839 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.649336 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.649611 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.649655 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.650269 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:23:58.650480 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.650485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.651401 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.651402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.652799 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.652802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.653265 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.653265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.653413 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.654026 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:23:58.654036 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:23:58.655669 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:23:58.657706 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.657715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.657717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.658025 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.658255 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:23:58.658256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:23:58.658256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:23:58.658419 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:23:58.658717 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:23:58.658725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:23:58.658892 (+ 167us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.659006 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.659183 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260810 15:23:58.659511 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.648677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:40051 (local address 127.4.64.126:34487)
0810 15:23:58.648749 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.648752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.648937 (+ 185us) server_negotiation.cc:407] Connection header received
0810 15:23:58.649450 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.649451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.649463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.649528 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:58.650566 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.651289 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.652891 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.653181 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.653253 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.655789 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:58.655791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:58.655791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:58.655800 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:58.657560 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.658110 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.658111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.658112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.658185 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.658529 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.658530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.658530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.658644 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:58.658705 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.659071 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.659420 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:23:58.660065 23403 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34487
I20260810 15:23:58.660202 23403 heartbeater.cc:464] Registering TS with master...
I20260810 15:23:58.660379 23403 heartbeater.cc:511] Master 127.4.64.126:34487 requested a full tablet report, sending...
I20260810 15:23:58.660815 22880 ts_manager.cc:194] Registered new tserver with Master: df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813)
I20260810 15:23:58.661173 22880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:40051
I20260810 15:23:58.671270 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:23:58.674202 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.671692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.4.64.126:34487)
0810 15:23:58.671760 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.671762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.671895 (+ 133us) server_negotiation.cc:407] Connection header received
0810 15:23:58.671958 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.671959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.671999 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.672065 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:58.672190 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.672696 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.673478 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.673738 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.673803 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.673950 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:58.674011 (+ 61us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 15:23:58.674146 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45690: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 15:23:58.674299 23104 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45690: token expired
W20260810 15:23:58.674541 22798 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.64.126:34487)
I20260810 15:23:58.680371 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.674838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.4.64.126:34487)
0810 15:23:58.674873 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.674875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.674883 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:23:58.674931 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.674932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.674939 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.674983 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:23:58.675450 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.675885 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.676585 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.676757 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.676817 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.678042 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:23:58.678043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:23:58.678043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:23:58.678050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:23:58.679359 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.679676 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.679677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.679677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.679732 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:23:58.679921 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:23:58.679922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:23:58.679922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:23:58.680043 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:23:58.680125 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.680210 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.680305 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:23:58.681752 22880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:23:58.683426 22880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:23:58.686003 23016 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd1ced1124043068d5a251f8df122ad (DEFAULT_TABLE table=test-table [id=b35cb79e548740889dadfd6e13fb1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:58.686252 23016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd1ced1124043068d5a251f8df122ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:58.687929 22973 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Bootstrap starting.
I20260810 15:23:58.688362 22973 tablet_bootstrap.cc:654] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:58.688954 22973 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: No bootstrap required, opened a new log
I20260810 15:23:58.689021 22973 ts_tablet_manager.cc:1397] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:23:58.689427 22973 raft_consensus.cc:348] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.689538 22973 raft_consensus.cc:374] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:58.689589 22973 raft_consensus.cc:729] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1363a9e45e746b9884ca5c089a93802, State: Initialized, Role: FOLLOWER
I20260810 15:23:58.689666 22973 consensus_queue.cc:260] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.689770 22973 ts_tablet_manager.cc:1428] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:23:58.691020 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.685646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:45525 (local address 127.0.0.1:57504)
0810 15:23:58.685836 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.685849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.685898 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.686258 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.686260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.686262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:58.686330 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.686331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.687566 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.687568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.688966 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.688970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.690700 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.690700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.690798 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.690823 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.690894 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.690948 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260810 15:23:58.691054 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.685849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.4.64.66:45525)
0810 15:23:58.686034 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.686040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.686051 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:23:58.686099 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.686118 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.686135 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.686427 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:58.686518 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.687469 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.689843 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.690622 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.690701 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.690772 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.690914 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.690965 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260810 15:23:58.691789 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.685636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:40813 (local address 127.0.0.1:57318)
0810 15:23:58.687939 (+ 2303us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.687952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.688016 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.688257 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.688259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.688261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:58.688305 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.688307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.689551 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.689552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.690701 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.690705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.691590 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.691591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.691643 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.691647 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.691693 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.691728 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2257,"thread_start_us":101,"threads_started":1}
I20260810 15:23:58.691831 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.685999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.4.64.67:40813)
0810 15:23:58.686254 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.686259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.688082 (+ 1823us) server_negotiation.cc:407] Connection header received
0810 15:23:58.688135 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.688138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.688155 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.688239 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:58.688377 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.689465 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.690787 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.691508 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.691588 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.691642 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.691707 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.691745 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":121,"threads_started":1}
I20260810 15:23:58.691872 23182 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd1ced1124043068d5a251f8df122ad (DEFAULT_TABLE table=test-table [id=b35cb79e548740889dadfd6e13fb1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:58.692236 23182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd1ced1124043068d5a251f8df122ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:58.694386 23330 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd1ced1124043068d5a251f8df122ad (DEFAULT_TABLE table=test-table [id=b35cb79e548740889dadfd6e13fb1d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:23:58.694753 23330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd1ced1124043068d5a251f8df122ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:23:58.695631 23416 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Bootstrap starting.
I20260810 15:23:58.696221 23416 tablet_bootstrap.cc:654] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:58.696861 23416 log.cc:824] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:58.697769 23416 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: No bootstrap required, opened a new log
I20260810 15:23:58.697829 23416 ts_tablet_manager.cc:1397] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:58.698342 23419 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Bootstrap starting.
I20260810 15:23:58.698964 23419 tablet_bootstrap.cc:654] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Neither blocks nor log segments found. Creating new log.
I20260810 15:23:58.699501 23419 log.cc:824] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Log is configured to *not* fsync() on all Append() calls
I20260810 15:23:58.699450 23416 raft_consensus.cc:348] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.699630 23416 raft_consensus.cc:374] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:58.699680 23416 raft_consensus.cc:729] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4818ba371a1b40e296a97d2fa5ca33d0, State: Initialized, Role: FOLLOWER
I20260810 15:23:58.699818 23416 consensus_queue.cc:260] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.700141 23254 heartbeater.cc:503] Master 127.4.64.126:34487 was elected leader, sending a full tablet report...
I20260810 15:23:58.700124 23416 ts_tablet_manager.cc:1428] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:23:58.700313 23419 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: No bootstrap required, opened a new log
I20260810 15:23:58.700390 23419 ts_tablet_manager.cc:1397] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:23:58.702243 23419 raft_consensus.cc:348] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.702409 23419 raft_consensus.cc:374] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:23:58.702456 23419 raft_consensus.cc:729] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df61a2c31df7499d8d294330d2f358af, State: Initialized, Role: FOLLOWER
I20260810 15:23:58.702581 23419 consensus_queue.cc:260] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.702891 23419 ts_tablet_manager.cc:1428] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:23:58.702912 23403 heartbeater.cc:503] Master 127.4.64.126:34487 was elected leader, sending a full tablet report...
W20260810 15:23:58.765031 23255 tablet.cc:2369] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:23:58.855145 23102 tablet.cc:2369] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:58.880488 23422 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:23:58.880640 23422 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.881119 23422 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525), d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931)
I20260810 15:23:58.885279 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.881374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:45525 (local address 127.4.64.67:49865)
0810 15:23:58.881465 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.881495 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.881575 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.881831 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.881832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.881835 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:58.881919 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.881921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.882898 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.882899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.884213 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.884216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.884969 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.884970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.885054 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.885060 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.885132 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.885182 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":26}
I20260810 15:23:58.885277 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.881449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:49865 (local address 127.4.64.66:45525)
0810 15:23:58.881518 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.881521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.881552 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:23:58.881691 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.881692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.881706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.881795 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:58.882000 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.882812 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.884339 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.884888 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.884965 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.884979 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.885158 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.885206 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:23:58.885309 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.881664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:55957 (local address 127.4.64.65:44931)
0810 15:23:58.881752 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.881754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.881877 (+ 123us) server_negotiation.cc:407] Connection header received
0810 15:23:58.881931 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.881931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.881974 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.882050 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:23:58.882186 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.882914 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.884308 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.884998 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.885064 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.885181 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.885201 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.885243 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 15:23:58.885298 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.881556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44931 (local address 127.4.64.67:55957)
0810 15:23:58.881776 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.881803 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 15:23:58.881870 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:23:58.882060 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:23:58.882060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:23:58.882062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:23:58.882112 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.882113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.882987 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.882988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.884213 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:23:58.884216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.885072 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.885073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:23:58.885154 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.885157 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:23:58.885198 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 15:23:58.885229 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":25,"thread_start_us":143,"threads_started":1}
I20260810 15:23:58.885728 23040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd1ced1124043068d5a251f8df122ad" candidate_uuid: "df61a2c31df7499d8d294330d2f358af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1363a9e45e746b9884ca5c089a93802" is_pre_election: true
I20260810 15:23:58.885950 23040 raft_consensus.cc:2454] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df61a2c31df7499d8d294330d2f358af in term 0.
I20260810 15:23:58.885895 23202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd1ced1124043068d5a251f8df122ad" candidate_uuid: "df61a2c31df7499d8d294330d2f358af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" is_pre_election: true
I20260810 15:23:58.886065 23202 raft_consensus.cc:2454] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df61a2c31df7499d8d294330d2f358af in term 0.
I20260810 15:23:58.886220 23277 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1363a9e45e746b9884ca5c089a93802, df61a2c31df7499d8d294330d2f358af; no voters:
I20260810 15:23:58.886368 23422 raft_consensus.cc:2781] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:23:58.886462 23422 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:23:58.886499 23422 raft_consensus.cc:3037] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:58.887238 23422 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.887405 23422 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [CANDIDATE]: Term 1 election: Requested vote from peers 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525), d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931)
I20260810 15:23:58.887600 23202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd1ced1124043068d5a251f8df122ad" candidate_uuid: "df61a2c31df7499d8d294330d2f358af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4818ba371a1b40e296a97d2fa5ca33d0"
I20260810 15:23:58.887598 23040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd1ced1124043068d5a251f8df122ad" candidate_uuid: "df61a2c31df7499d8d294330d2f358af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1363a9e45e746b9884ca5c089a93802"
I20260810 15:23:58.887694 23040 raft_consensus.cc:3037] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:58.887701 23202 raft_consensus.cc:3037] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:23:58.888449 23202 raft_consensus.cc:2454] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df61a2c31df7499d8d294330d2f358af in term 1.
I20260810 15:23:58.888461 23040 raft_consensus.cc:2454] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df61a2c31df7499d8d294330d2f358af in term 1.
I20260810 15:23:58.888626 23277 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4818ba371a1b40e296a97d2fa5ca33d0, df61a2c31df7499d8d294330d2f358af; no voters:
I20260810 15:23:58.888715 23422 raft_consensus.cc:2781] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:23:58.888870 23422 raft_consensus.cc:686] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 1 LEADER]: Becoming Leader. State: Replica: df61a2c31df7499d8d294330d2f358af, State: Running, Role: LEADER
I20260810 15:23:58.888973 23422 consensus_queue.cc:237] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:23:58.889937 22879 catalog_manager.cc:5495] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af reported cstate change: term changed from 0 to 1, leader changed from <none> to df61a2c31df7499d8d294330d2f358af (127.4.64.67). New cstate: current_term: 1 leader_uuid: "df61a2c31df7499d8d294330d2f358af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } health_report { overall_health: UNKNOWN } } }
W20260810 15:23:58.898463 23404 tablet.cc:2369] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:23:58.920387 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.917819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.4.64.67:40813)
0810 15:23:58.917884 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:23:58.917887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:23:58.917899 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:23:58.917965 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:23:58.917966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:23:58.917986 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:23:58.918065 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:23:58.918223 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.918758 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.919518 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:23:58.919720 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:23:58.919801 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:23:58.919987 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:23:58.920105 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:23:58.920212 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 15:23:58.920232 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 15:23:58.920254 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:23:58.922557 23040 raft_consensus.cc:1261] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Refusing update from remote peer df61a2c31df7499d8d294330d2f358af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:58.922660 23202 raft_consensus.cc:1261] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Refusing update from remote peer df61a2c31df7499d8d294330d2f358af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:23:58.922940 23422 consensus_queue.cc:1035] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:58.923075 23424 consensus_queue.cc:1035] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [LEADER]: Connected to new peer: Peer: permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:23:58.925918 23431 mvcc.cc:205] Tried to move back new op lower bound from 7316993797823713280 to 7316993797690138624. Current Snapshot: MvccSnapshot[applied={T|T < 7316993797823713280}]
I20260810 15:23:58.926023 23430 mvcc.cc:205] Tried to move back new op lower bound from 7316993797823713280 to 7316993797690138624. Current Snapshot: MvccSnapshot[applied={T|T < 7316993797823713280}]
I20260810 15:23:58.928225 22878 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:58.928347 22878 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:23:58.929147 22878 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Sending DeleteTablet for 3 replicas of tablet 8dd1ced1124043068d5a251f8df122ad
I20260810 15:23:58.929454 23330 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dd1ced1124043068d5a251f8df122ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:58 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57318
I20260810 15:23:58.929492 23016 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dd1ced1124043068d5a251f8df122ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:58 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:54338
I20260810 15:23:58.929504 23182 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dd1ced1124043068d5a251f8df122ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:23:58 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:57504
I20260810 15:23:58.929713 23440 tablet_replica.cc:330] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: stopping tablet replica
I20260810 15:23:58.929775 23440 raft_consensus.cc:2229] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:58.929831 23440 raft_consensus.cc:2258] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:58.929894 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 22926
I20260810 15:23:58.935695 23439 tablet_replica.cc:330] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: stopping tablet replica
I20260810 15:23:58.935905 23439 raft_consensus.cc:2229] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:23:58.936112 23439 raft_consensus.cc:2258] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:58.937320 23441 tablet_replica.cc:330] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: stopping tablet replica
I20260810 15:23:58.937515 23441 raft_consensus.cc:2229] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:23:58.937714 23441 raft_consensus.cc:2258] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:23:58.938351 23441 ts_tablet_manager.cc:1905] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:58.940593 23441 ts_tablet_manager.cc:1918] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:58.940829 23441 log.cc:1196] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/8dd1ced1124043068d5a251f8df122ad
I20260810 15:23:58.941221 23441 ts_tablet_manager.cc:1939] T 8dd1ced1124043068d5a251f8df122ad P 4818ba371a1b40e296a97d2fa5ca33d0: Deleting consensus metadata
W20260810 15:23:58.941663 22864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:44931 (error 108)
I20260810 15:23:58.941941 22864 catalog_manager.cc:4841] TS 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): tablet 8dd1ced1124043068d5a251f8df122ad (table test-table [id=b35cb79e548740889dadfd6e13fb1d98]) successfully deleted
I20260810 15:23:58.942194 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 23105
I20260810 15:23:58.942364 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:23:58.942204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:44931 (local address 127.0.0.1:54342)
0810 15:23:58.942282 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 15:23:58.942324 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:44931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 15:23:58.944216 23439 ts_tablet_manager.cc:1905] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:23:58.950449 23439 ts_tablet_manager.cc:1918] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:23:58.950543 23439 log.cc:1196] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/8dd1ced1124043068d5a251f8df122ad
I20260810 15:23:58.950863 23439 ts_tablet_manager.cc:1939] T 8dd1ced1124043068d5a251f8df122ad P df61a2c31df7499d8d294330d2f358af: Deleting consensus metadata
W20260810 15:23:58.950760 22864 catalog_manager.cc:4584] TS d1363a9e45e746b9884ca5c089a93802 (127.4.64.65:44931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8dd1ced1124043068d5a251f8df122ad: Network error: Client connection negotiation failed: client connection to 127.4.64.65:44931: connect: Connection refused (error 111)
I20260810 15:23:58.951344 22863 catalog_manager.cc:4841] TS df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): tablet 8dd1ced1124043068d5a251f8df122ad (table test-table [id=b35cb79e548740889dadfd6e13fb1d98]) successfully deleted
I20260810 15:23:58.955678 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 23258
I20260810 15:23:58.963120 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 22843
I20260810 15:24:18.971942 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34487
--webserver_interface=127.4.64.126
--webserver_port=41825
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:19.061342 23444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:19.061563 23444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:19.061587 23444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:19.062844 23444 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:24:19.062891 23444 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:24:19.062904 23444 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:24:19.062915 23444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:19.062969 23444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:19.062997 23444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:19.063025 23444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:19.063076 23444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:19.064697 23444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=41825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:19.064980 23444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:19.065258 23444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:19.070547 23451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.070616 23450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.070564 23453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:19.070914 23444 server_base.cc:1034] running on GCE node
I20260810 15:24:19.071177 23444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:19.071482 23444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:19.072665 23444 hybrid_clock.cc:648] HybridClock initialized: now 1786375459072641 us; error 53 us; skew 500 ppm
Aug 10 15:24:19 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:19.075469 23444 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:24:19.075922 23444 webserver.cc:460] Webserver started at http://127.4.64.126:41825/ using document root <none> and password file <none>
I20260810 15:24:19.076184 23444 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:19.076256 23444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:19.077761 23444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:19.078501 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.078706 23444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:19.078819 23444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "cd7559e88849428295cd69973ee5f7de"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "523c45a7267d03c5dff1b72d69a4dfc2"
server_key_iv: "ab9904f3467ae65255392efb4ca59219"
server_key_version: "encryptionkey@0"
I20260810 15:24:19.079254 23444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:19.090756 23444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:19.092741 23444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:19.092914 23444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:19.100502 23444 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34487
I20260810 15:24:19.100538 23517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34487 every 8 connection(s)
I20260810 15:24:19.100961 23444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:19.102669 23518 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:24:19.103380 23518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap starting.
I20260810 15:24:19.108906 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 23444
I20260810 15:24:19.109359 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:44931
--local_ip_for_outbound_sockets=127.4.64.65
--tserver_master_addrs=127.4.64.126:34487
--webserver_port=43903
--webserver_interface=127.4.64.65
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:24:19.113281 23518 log.cc:824] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:19.115993 23518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:24:19.116384 23518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Bootstrap complete.
I20260810 15:24:19.118949 23518 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:24:19.119194 23518 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7559e88849428295cd69973ee5f7de, State: Initialized, Role: FOLLOWER
I20260810 15:24:19.119444 23518 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:24:19.119521 23518 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:19.119553 23518 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:19.119628 23518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:24:19.120713 23518 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:24:19.120848 23518 leader_election.cc:302] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7559e88849428295cd69973ee5f7de; no voters:
I20260810 15:24:19.121088 23518 leader_election.cc:288] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 15:24:19.121135 23523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:24:19.121268 23523 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [term 3 LEADER]: Becoming Leader. State: Replica: cd7559e88849428295cd69973ee5f7de, State: Running, Role: LEADER
I20260810 15:24:19.121392 23518 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:19.121369 23523 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } }
I20260810 15:24:19.121685 23523 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7559e88849428295cd69973ee5f7de. Latest consensus state: current_term: 3 leader_uuid: "cd7559e88849428295cd69973ee5f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } } }
I20260810 15:24:19.121767 23523 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:19.121690 23524 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cd7559e88849428295cd69973ee5f7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7559e88849428295cd69973ee5f7de" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34487 } } }
I20260810 15:24:19.121944 23524 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:19.122097 23527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:19.123212 23527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:19.123373 23527 catalog_manager.cc:1259] Loaded cluster ID: 85c8f11b136444f2bc4a39d98d41e5c5
I20260810 15:24:19.123797 23527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:19.125240 23527 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:19.125464 23527 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Loaded TSK: 0
I20260810 15:24:19.125537 23527 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd7559e88849428295cd69973ee5f7de: Loaded TSK: 1
F20260810 15:24:19.125859 23527 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd7a10b79ad google::LogMessage::Fail() at ??:0
@ 0x7fd7a10ba88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd7a10b74de google::LogMessage::Flush() at ??:0
@ 0x7fd7a10b8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd7a3824104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd7a3821d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd7a90111cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd7a901148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd7a9011594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd7a8fd7c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd7a8fe1ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd7a2f850d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd7a2f7a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd7a3dde6db start_thread at ??:0
@ 0x7fd79fbb561f clone at ??:0
W20260810 15:24:19.238895 23521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:19.239204 23521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:19.239256 23521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:19.241154 23521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:19.241242 23521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:19.241291 23521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:19.243726 23521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:44931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=43903
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:19.244014 23521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:19.244313 23521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:19.250376 23566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.250556 23568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.250375 23564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:19.250923 23521 server_base.cc:1034] running on GCE node
I20260810 15:24:19.251196 23521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:19.251479 23521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:19.252664 23521 hybrid_clock.cc:648] HybridClock initialized: now 1786375459252647 us; error 35 us; skew 500 ppm
Aug 10 15:24:19 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:19.255735 23521 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:24:19.256263 23521 webserver.cc:460] Webserver started at http://127.4.64.65:43903/ using document root <none> and password file <none>
I20260810 15:24:19.256619 23521 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:19.256697 23521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:19.258332 23521 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:24:19.259150 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.259400 23521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:19.259526 23521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "d1363a9e45e746b9884ca5c089a93802"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "0f7234b5fb872395c0e14d2f2dad3bb3"
server_key_iv: "580e4c9ad6312608ba912ddbf7222d3c"
server_key_version: "encryptionkey@0"
I20260810 15:24:19.259932 23521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:19.278757 23521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:19.281193 23521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:19.281371 23521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:19.281694 23521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:19.282191 23585 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:24:19.284262 23521 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:24:19.284351 23521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:24:19.284400 23521 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:24:19.285193 23521 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:24:19.285260 23521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:24:19.285336 23585 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Bootstrap starting.
I20260810 15:24:19.292969 23521 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:44931
I20260810 15:24:19.293151 23706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:44931 every 8 connection(s)
I20260810 15:24:19.293453 23521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:24:19.297194 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 23521
I20260810 15:24:19.297596 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:45525
--local_ip_for_outbound_sockets=127.4.64.66
--tserver_master_addrs=127.4.64.126:34487
--webserver_port=35065
--webserver_interface=127.4.64.66
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:24:19.297708 23585 log.cc:824] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:19.300591 23585 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:24:19.300947 23585 tablet_bootstrap.cc:492] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Bootstrap complete.
I20260810 15:24:19.302386 23585 ts_tablet_manager.cc:1397] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.010s
I20260810 15:24:19.303862 23585 raft_consensus.cc:348] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:19.304208 23585 raft_consensus.cc:729] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1363a9e45e746b9884ca5c089a93802, State: Initialized, Role: FOLLOWER
I20260810 15:24:19.304414 23585 consensus_queue.cc:260] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:19.304757 23585 ts_tablet_manager.cc:1428] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260810 15:24:19.363493 23710 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:24:19.294443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:58219)
0810 15:24:19.294669 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.294710 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:19.295237 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:19.363282 (+ 68045us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
W20260810 15:24:19.363795 23707 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:34487 (0 consecutive failures): Network error: Failed to ping master at 127.4.64.126:34487: Client connection negotiation failed: client connection to 127.4.64.126:34487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:24:19.364171 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.364044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:39147)
0810 15:24:19.364100 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.364149 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 15:24:19.364568 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.364472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:42993)
0810 15:24:19.364519 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.364544 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:24:19.364678 23707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:24:19.388454 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:19.388701 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:19.388742 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:19.390087 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:19.390154 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:19.390202 23713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:19.391683 23713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:45525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=35065
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:19.391961 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:19.392259 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:19.397593 23719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.397563 23720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.397559 23722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:19.397851 23713 server_base.cc:1034] running on GCE node
I20260810 15:24:19.398037 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:19.398239 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:19.399389 23713 hybrid_clock.cc:648] HybridClock initialized: now 1786375459399376 us; error 27 us; skew 500 ppm
Aug 10 15:24:19 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:19.402493 23713 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:24:19.402927 23713 webserver.cc:460] Webserver started at http://127.4.64.66:35065/ using document root <none> and password file <none>
I20260810 15:24:19.403116 23713 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:19.403162 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:19.404599 23713 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:19.405416 23730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.405679 23713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:19.405803 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "4818ba371a1b40e296a97d2fa5ca33d0"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "5aa493b0da0e871513fc0fe9b2b66750"
server_key_iv: "f8b41b9959fefc04182fd4c9f4c31dc3"
server_key_version: "encryptionkey@0"
I20260810 15:24:19.406162 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:19.422032 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:19.424702 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:19.424894 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:19.425169 23713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:19.425535 23713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:19.425591 23713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.425642 23713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:19.425675 23713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.432825 23713 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:45525
I20260810 15:24:19.432871 23859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:45525 every 8 connection(s)
I20260810 15:24:19.433269 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:19.434554 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.434039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:50829)
0810 15:24:19.434244 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.434399 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
W20260810 15:24:19.434978 23860 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:34487 (0 consecutive failures): Network error: Failed to ping master at 127.4.64.126:34487: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
I20260810 15:24:19.435097 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 23713
I20260810 15:24:19.435304 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.435211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:33393)
0810 15:24:19.435249 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.435285 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260810 15:24:19.435510 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:40813
--local_ip_for_outbound_sockets=127.4.64.67
--tserver_master_addrs=127.4.64.126:34487
--webserver_port=39233
--webserver_interface=127.4.64.67
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:41715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:24:19.435582 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.435499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:49683)
0810 15:24:19.435542 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.435563 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:24:19.435683 23860 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:24:19.523617 23863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:19.523873 23863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:19.523914 23863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:19.525198 23863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:19.525271 23863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:19.525316 23863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:19.526847 23863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:40813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=39233
--tserver_master_addrs=127.4.64.126:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:19.527127 23863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:19.527382 23863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:19.532770 23869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:19.532770 23871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:19.532951 23863 server_base.cc:1034] running on GCE node
W20260810 15:24:19.533072 23868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:19.533300 23863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:19.533583 23863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:19.534785 23863 hybrid_clock.cc:648] HybridClock initialized: now 1786375459534763 us; error 41 us; skew 500 ppm
Aug 10 15:24:19 dist-test-slave-9zdj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:19.537415 23863 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:24:19.537945 23863 webserver.cc:460] Webserver started at http://127.4.64.67:39233/ using document root <none> and password file <none>
I20260810 15:24:19.538144 23863 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:19.538193 23863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:19.539559 23863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:19.540362 23881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.540580 23863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:19.540673 23863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "df61a2c31df7499d8d294330d2f358af"
format_stamp: "Formatted at 2026-08-10 15:23:37 on dist-test-slave-9zdj"
server_key: "22eee2fd8345a9088cb136319ecf08cb"
server_key_iv: "350c913090ed5b6d331baf2d29b32003"
server_key_version: "encryptionkey@0"
I20260810 15:24:19.541046 23863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:19.561792 23863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:19.564338 23863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:19.564468 23863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:19.564680 23863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:19.565032 23863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:19.565071 23863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.565102 23863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:19.565130 23863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:19.571447 23863 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:40813
I20260810 15:24:19.571532 24009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:40813 every 8 connection(s)
I20260810 15:24:19.571919 23863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:19.572245 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 23863
I20260810 15:24:19.573431 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.572695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:48841)
0810 15:24:19.572934 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.573047 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
W20260810 15:24:19.573679 24010 heartbeater.cc:644] Failed to heartbeat to 127.4.64.126:34487 (0 consecutive failures): Network error: Failed to ping master at 127.4.64.126:34487: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
I20260810 15:24:19.573973 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.573869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:35715)
0810 15:24:19.573920 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.573953 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:24:19.574249 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:19.574162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:45863)
0810 15:24:19.574205 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:19.574227 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:24:19.574359 24010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:24:20.365540 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:20.365066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:50067)
0810 15:24:20.365337 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:20.365445 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260810 15:24:20.436242 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:20.435989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:35785)
0810 15:24:20.436127 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:20.436172 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260810 15:24:20.574961 24016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:20.574726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:42177)
0810 15:24:20.574859 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:20.574901 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260810 15:24:21.053588 24017 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:21.053785 24017 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:21.054229 24017 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813), 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
Aug 10 15:24:21 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.66@KRBTEST.COM
Aug 10 15:24:21 dist-test-slave-9zdj krb5kdc[22203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.67@KRBTEST.COM
I20260810 15:24:21.066427 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.054683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:58659 (local address 127.4.64.66:45525)
0810 15:24:21.054810 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:21.054818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:21.054835 (+ 17us) server_negotiation.cc:407] Connection header received
0810 15:24:21.055444 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:21.055447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:21.055494 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:21.055655 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:21.056652 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.058119 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.059663 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.060069 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.060202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:21.062623 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:21.062628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:21.062628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:21.062640 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:21.064569 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:21.065051 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:21.065055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:21.065055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:21.065171 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:21.065462 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:21.065462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:21.065463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:21.065785 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:21.065879 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:21.066166 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 15:24:21.066306 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":51,"threads_started":1}
I20260810 15:24:21.066417 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.054409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:45525 (local address 127.4.64.65:58659)
0810 15:24:21.054544 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:21.054556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:21.054621 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:21.055693 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:21.055696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:21.056272 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:21.056520 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:21.056528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.058216 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.058217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:21.059566 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:21.059571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.060170 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.060171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:21.060290 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:21.060858 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:21.060876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:21.062488 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:21.064684 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:21.064692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:21.064694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:21.064963 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:21.065261 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:21.065262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:21.065262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:21.065382 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:21.065858 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:21.065865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:21.066002 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 15:24:21.066113 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 15:24:21.066319 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260810 15:24:21.066555 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.054522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:45111 (local address 127.4.64.67:40813)
0810 15:24:21.054707 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:21.054720 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:21.054760 (+ 40us) server_negotiation.cc:407] Connection header received
0810 15:24:21.055467 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:21.055471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:21.055526 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:21.055663 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:21.056645 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.058117 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.060228 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.060628 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.060750 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:21.063044 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:21.063048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:21.063049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:21.063061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:21.064726 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:21.065145 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:21.065148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:21.065149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:21.065249 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:21.065464 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:21.065465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:21.065465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:21.065856 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:21.065965 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:21.066281 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 15:24:21.066438 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":54,"threads_started":1}
I20260810 15:24:21.066996 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.054416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:40813 (local address 127.4.64.65:45111)
0810 15:24:21.054667 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:21.054678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:21.054753 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:21.055777 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:21.055778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:21.056265 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:21.056520 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:21.056528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.059668 (+ 3140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.059668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:21.060137 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:21.060138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:21.060725 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:21.060726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:21.060780 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:21.061182 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:21.061186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:21.062926 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:21.064828 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:21.064829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:21.064830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:21.065036 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:21.065330 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:21.065331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:21.065331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:21.065400 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:21.065946 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:21.065947 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:21.066007 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 15:24:21.066119 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 15:24:21.066423 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":124,"thread_start_us":53,"threads_started":1}
W20260810 15:24:21.067598 23579 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
W20260810 15:24:21.067799 23577 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
I20260810 15:24:21.067875 23577 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1363a9e45e746b9884ca5c089a93802; no voters: 4818ba371a1b40e296a97d2fa5ca33d0, df61a2c31df7499d8d294330d2f358af
I20260810 15:24:21.068019 24017 raft_consensus.cc:2726] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:24:21.366405 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.366220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:43209)
0810 15:24:21.366312 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:21.366362 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 15:24:21.437122 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.436778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:58927)
0810 15:24:21.436997 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:21.437059 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260810 15:24:21.575750 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:21.575516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:52747)
0810 15:24:21.575656 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:21.575701 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260810 15:24:22.367319 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:22.367044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:49003)
0810 15:24:22.367216 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:22.367271 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260810 15:24:22.438194 24027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:22.437966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:36743)
0810 15:24:22.438102 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:22.438146 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260810 15:24:22.576633 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:22.576352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:36943)
0810 15:24:22.576538 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:22.576584 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
I20260810 15:24:22.900453 24029 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:22.900561 24029 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:22.900815 24029 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813), 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
W20260810 15:24:22.901492 23577 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
W20260810 15:24:22.901584 23579 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
I20260810 15:24:22.901701 23579 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1363a9e45e746b9884ca5c089a93802; no voters: 4818ba371a1b40e296a97d2fa5ca33d0, df61a2c31df7499d8d294330d2f358af
I20260810 15:24:22.901787 24029 raft_consensus.cc:2726] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:24:23.368283 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:23.367996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:38555)
0810 15:24:23.368171 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:23.368214 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260810 15:24:23.439249 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:23.438909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:37601)
0810 15:24:23.439066 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:23.439111 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260810 15:24:23.577447 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:23.577206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:46133)
0810 15:24:23.577351 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:23.577399 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260810 15:24:24.369133 24034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:24.368867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:35215)
0810 15:24:24.369032 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:24.369078 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260810 15:24:24.440117 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:24.439874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:49841)
0810 15:24:24.440020 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:24.440069 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260810 15:24:24.543442 24036 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:24.543552 24036 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:24.543816 24036 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813), 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
W20260810 15:24:24.544307 23577 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
W20260810 15:24:24.544410 23579 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
I20260810 15:24:24.544474 23579 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1363a9e45e746b9884ca5c089a93802; no voters: 4818ba371a1b40e296a97d2fa5ca33d0, df61a2c31df7499d8d294330d2f358af
I20260810 15:24:24.544562 24036 raft_consensus.cc:2726] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:24:24.578276 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:24.578015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:42793)
0810 15:24:24.578175 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:24.578227 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":114,"threads_started":1}
I20260810 15:24:25.370007 24038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:25.369760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:59621)
0810 15:24:25.369910 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:25.369957 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260810 15:24:25.440927 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:25.440680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:49215)
0810 15:24:25.440811 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:25.440878 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260810 15:24:25.579110 24040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:25.578817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:34323)
0810 15:24:25.579002 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:25.579058 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260810 15:24:26.370782 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:26.370513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:51235)
0810 15:24:26.370684 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:26.370732 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260810 15:24:26.441809 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:26.441492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:59639)
0810 15:24:26.441722 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:26.441764 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260810 15:24:26.579880 24046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:26.579619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:44399)
0810 15:24:26.579789 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:26.579832 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260810 15:24:27.221056 24047 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:27.221167 24047 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:27.221459 24047 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813), 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
W20260810 15:24:27.221873 23579 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
W20260810 15:24:27.221959 23577 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
I20260810 15:24:27.222024 23577 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1363a9e45e746b9884ca5c089a93802; no voters: 4818ba371a1b40e296a97d2fa5ca33d0, df61a2c31df7499d8d294330d2f358af
I20260810 15:24:27.222097 24047 raft_consensus.cc:2726] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:24:27.371737 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:27.371482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:57039)
0810 15:24:27.371638 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:27.371679 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260810 15:24:27.442637 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:27.442369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:46591)
0810 15:24:27.442540 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:27.442590 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260810 15:24:27.580704 24050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:27.580461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:51909)
0810 15:24:27.580619 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:27.580660 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260810 15:24:28.372601 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:28.372373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:46565)
0810 15:24:28.372508 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:28.372550 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260810 15:24:28.443435 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:28.443146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:35045)
0810 15:24:28.443285 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:28.443372 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260810 15:24:28.581550 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:28.581249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:33057)
0810 15:24:28.581456 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:28.581501 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260810 15:24:29.373623 24054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:29.373297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:45831)
0810 15:24:29.373490 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:29.373537 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260810 15:24:29.444288 24055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:29.444051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:35569)
0810 15:24:29.444190 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:29.444238 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260810 15:24:29.582453 24056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:29.582220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:47241)
0810 15:24:29.582357 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:29.582403 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260810 15:24:30.374563 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:30.374285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:55931)
0810 15:24:30.374464 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:30.374511 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260810 15:24:30.445326 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:30.444941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:33653)
0810 15:24:30.445115 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:30.445163 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260810 15:24:30.583388 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:30.583125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:53219)
0810 15:24:30.583274 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:30.583317 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260810 15:24:31.375449 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:31.375179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:59667)
0810 15:24:31.375348 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:31.375399 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260810 15:24:31.446220 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:31.445978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:52605)
0810 15:24:31.446130 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:31.446170 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260810 15:24:31.584052 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:31.583796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:38199)
0810 15:24:31.583955 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:31.584000 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260810 15:24:32.376293 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:32.376022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:58897)
0810 15:24:32.376198 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:32.376243 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260810 15:24:32.446975 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:32.446725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:41045)
0810 15:24:32.446883 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:32.446927 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260810 15:24:32.585135 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:32.584862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:42777)
0810 15:24:32.585028 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:32.585072 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260810 15:24:33.377010 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:33.376760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:59543)
0810 15:24:33.376921 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:33.376959 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260810 15:24:33.447778 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:33.447491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:43429)
0810 15:24:33.447672 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:33.447727 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260810 15:24:33.586032 24069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:33.585749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:34075)
0810 15:24:33.585927 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:33.585974 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260810 15:24:34.006827 24070 raft_consensus.cc:482] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:34.006933 24070 raft_consensus.cc:504] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df61a2c31df7499d8d294330d2f358af" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 40813 } } peers { permanent_uuid: "4818ba371a1b40e296a97d2fa5ca33d0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 45525 } } peers { permanent_uuid: "d1363a9e45e746b9884ca5c089a93802" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 44931 } }
I20260810 15:24:34.007228 24070 leader_election.cc:288] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813), 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525)
W20260810 15:24:34.007568 23577 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df61a2c31df7499d8d294330d2f358af (127.4.64.67:40813): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
W20260810 15:24:34.007653 23579 leader_election.cc:341] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4818ba371a1b40e296a97d2fa5ca33d0 (127.4.64.66:45525): Not found: Tablet not found: 8dd1ced1124043068d5a251f8df122ad
I20260810 15:24:34.007682 23577 leader_election.cc:302] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1363a9e45e746b9884ca5c089a93802; no voters: 4818ba371a1b40e296a97d2fa5ca33d0, df61a2c31df7499d8d294330d2f358af
I20260810 15:24:34.007767 24070 raft_consensus.cc:2726] T 8dd1ced1124043068d5a251f8df122ad P d1363a9e45e746b9884ca5c089a93802 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:24:34.377892 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:34.377583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.65:60759)
0810 15:24:34.377803 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:34.377843 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":155,"threads_started":1}
I20260810 15:24:34.448714 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:34.448453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.66:40599)
0810 15:24:34.448606 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:34.448664 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260810 15:24:34.578985 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 23521
I20260810 15:24:34.587078 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:34.586832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34487 (local address 127.4.64.67:41565)
0810 15:24:34.586982 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:34.587028 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.126:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260810 15:24:34.589013 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 23713
I20260810 15:24:34.597566 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 23863
2026-08-10T15:24:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58737 ms)
[----------] 30 tests from SecurityITest (246098 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:34 dist-test-slave-9zdj krb5kdc[24077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:34 dist-test-slave-9zdj krb5kdc[24077](info): set up 2 sockets
Aug 10 15:24:34 dist-test-slave-9zdj krb5kdc[24077](info): commencing operation
krb5kdc: starting...
W20260810 15:24:35.734690 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:35 dist-test-slave-9zdj krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:24:35.877789 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37899
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:35.963294 24091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:35.963551 24091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:35.963591 24091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:35.964742 24091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:35.964818 24091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:35.964843 24091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:35.964875 24091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:35.964900 24091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:35.966564 24091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:37899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:35.966850 24091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:35.967090 24091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:24:35.972872 24091 server_base.cc:1034] running on GCE node
W20260810 15:24:35.972883 24097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:35.972918 24096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:35.973039 24099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:35.973585 24091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:35.973975 24091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:35.975178 24091 hybrid_clock.cc:648] HybridClock initialized: now 1786375475975164 us; error 49 us; skew 500 ppm
Aug 10 15:24:35 dist-test-slave-9zdj krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375475, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:35.977957 24091 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:24:35.978466 24091 webserver.cc:460] Webserver started at http://127.4.64.126:41365/ using document root <none> and password file <none>
I20260810 15:24:35.978688 24091 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:35.978729 24091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:35.978833 24091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:35.979879 24091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "53db8c287f874c1b913ccf4250558cfc"
format_stamp: "Formatted at 2026-08-10 15:24:35 on dist-test-slave-9zdj"
server_key: "e7dc79cd09c294372bd6eb3c07745f4b"
server_key_iv: "73fedfd6497e4c7cc1c9590bdf1210bb"
server_key_version: "encryptionkey@0"
I20260810 15:24:35.980192 24091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "53db8c287f874c1b913ccf4250558cfc"
format_stamp: "Formatted at 2026-08-10 15:24:35 on dist-test-slave-9zdj"
server_key: "e7dc79cd09c294372bd6eb3c07745f4b"
server_key_iv: "73fedfd6497e4c7cc1c9590bdf1210bb"
server_key_version: "encryptionkey@0"
I20260810 15:24:35.981583 24091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:24:35.982475 24106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:35.982693 24091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:35.982800 24091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "53db8c287f874c1b913ccf4250558cfc"
format_stamp: "Formatted at 2026-08-10 15:24:35 on dist-test-slave-9zdj"
server_key: "e7dc79cd09c294372bd6eb3c07745f4b"
server_key_iv: "73fedfd6497e4c7cc1c9590bdf1210bb"
server_key_version: "encryptionkey@0"
I20260810 15:24:35.982878 24091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:35.997121 24091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:35.999545 24091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:35.999704 24091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:36.003961 24091 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:37899
I20260810 15:24:36.003984 24162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:37899 every 8 connection(s)
I20260810 15:24:36.004477 24091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:36.005544 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:36.007939 24163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: Bootstrap starting.
I20260810 15:24:36.008555 24163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:36.008857 24163 log.cc:824] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:36.009837 24163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: No bootstrap required, opened a new log
I20260810 15:24:36.011605 24163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53db8c287f874c1b913ccf4250558cfc" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37899 } }
I20260810 15:24:36.011771 24163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:36.011842 24163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53db8c287f874c1b913ccf4250558cfc, State: Initialized, Role: FOLLOWER
I20260810 15:24:36.012018 24163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53db8c287f874c1b913ccf4250558cfc" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37899 } }
I20260810 15:24:36.012115 24163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:36.012162 24163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:36.012214 24163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:36.012742 24163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53db8c287f874c1b913ccf4250558cfc" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37899 } }
I20260810 15:24:36.012864 24163 leader_election.cc:302] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53db8c287f874c1b913ccf4250558cfc; no voters:
I20260810 15:24:36.013069 24163 leader_election.cc:288] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:36.013103 24167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:36.013273 24167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [term 1 LEADER]: Becoming Leader. State: Replica: 53db8c287f874c1b913ccf4250558cfc, State: Running, Role: LEADER
I20260810 15:24:36.013355 24163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:36.013425 24167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53db8c287f874c1b913ccf4250558cfc" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37899 } }
I20260810 15:24:36.013875 24169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53db8c287f874c1b913ccf4250558cfc. Latest consensus state: current_term: 1 leader_uuid: "53db8c287f874c1b913ccf4250558cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53db8c287f874c1b913ccf4250558cfc" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37899 } } }
I20260810 15:24:36.014024 24168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53db8c287f874c1b913ccf4250558cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53db8c287f874c1b913ccf4250558cfc" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 37899 } } }
I20260810 15:24:36.014290 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24091
I20260810 15:24:36.014416 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:36.014039 24169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:36.014676 4353 external_mini_cluster.cc:1428] Setting key cdf653e723e8be1d01fcc1162d5e7561
I20260810 15:24:36.014124 24168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc [sys.catalog]: This master's current role is: LEADER
W20260810 15:24:36.015478 24185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:24:36.015570 24185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:24:36.015636 24184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:36.015826 24184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:36.017269 24184 catalog_manager.cc:1347] Generated new cluster ID: c4c29150d72c4595b07f7fab97503863
I20260810 15:24:36.017326 24184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:36.026970 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.015400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.4.64.126:37899)
0810 15:24:36.015587 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.015599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.015641 (+ 42us) server_negotiation.cc:407] Connection header received
0810 15:24:36.015787 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.015795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.015856 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.016000 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:36.016781 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.018245 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.019391 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.019992 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.020110 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.023081 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:36.023086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:36.023086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:36.023101 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:36.025244 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.025661 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.025665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.025665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.025806 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.026016 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.026017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.026017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.026353 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:36.026521 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.026596 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.026748 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260810 15:24:36.033412 24184 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:36.034080 24184 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:36.036227 24184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: Generated new TSK 0
I20260810 15:24:36.036412 24184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:24:36.077032 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37899
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:36.185076 24192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:36.185356 24192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:36.185400 24192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:36.186811 24192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:36.186889 24192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:36.186935 24192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:36.188436 24192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:36.188726 24192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:36.188987 24192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:36.194784 24197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.194774 24200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.194784 24198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:36.195196 24192 server_base.cc:1034] running on GCE node
I20260810 15:24:36.195430 24192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:36.195664 24192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:36.196812 24192 hybrid_clock.cc:648] HybridClock initialized: now 1786375476196782 us; error 50 us; skew 500 ppm
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:36.199518 24192 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:24:36.199992 24192 webserver.cc:460] Webserver started at http://127.4.64.65:39913/ using document root <none> and password file <none>
I20260810 15:24:36.200260 24192 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:36.200336 24192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:36.200495 24192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:36.201673 24192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "57e5101758fd4ec99c9f3ca83be9277a"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "922e38ed551a246a099ac16ec590ea30"
server_key_iv: "c86bdfdfa95483949f16cf7654258705"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.202073 24192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "57e5101758fd4ec99c9f3ca83be9277a"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "922e38ed551a246a099ac16ec590ea30"
server_key_iv: "c86bdfdfa95483949f16cf7654258705"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.203583 24192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:24:36.204578 24207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.204957 24192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:36.205071 24192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "57e5101758fd4ec99c9f3ca83be9277a"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "922e38ed551a246a099ac16ec590ea30"
server_key_iv: "c86bdfdfa95483949f16cf7654258705"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.205170 24192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:36.213956 24192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:36.216746 24192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:36.216953 24192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:36.217247 24192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:36.217607 24192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:36.217700 24192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.217741 24192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:36.217770 24192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.223817 24192 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:39553
I20260810 15:24:36.223896 24330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:39553 every 8 connection(s)
I20260810 15:24:36.224318 24192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:36.234333 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24192
I20260810 15:24:36.234500 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:24:36.234603 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.225161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:45215 (local address 127.4.64.126:37899)
0810 15:24:36.225243 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.225246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.225450 (+ 204us) server_negotiation.cc:407] Connection header received
0810 15:24:36.226050 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.226051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.226066 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.226123 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:36.227117 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.227737 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.228851 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.229080 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.229145 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.231558 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:36.231559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:36.231560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:36.231566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:36.233000 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.233427 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.233428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.233428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.233486 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.233805 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.233806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.233806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.233922 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:36.233978 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.234390 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.234501 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 15:24:36.234728 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.225097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37899 (local address 127.4.64.65:45215)
0810 15:24:36.225322 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.225363 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.225941 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.226180 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.226183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.226803 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:36.227013 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.227018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.227868 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.227869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.228753 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.228756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.229186 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.229187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.229311 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.229914 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:36.229923 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:36.231471 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:36.233088 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:36.233095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:36.233097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:36.233354 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:36.233570 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:36.233571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:36.233571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:36.233737 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:36.234001 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:36.234008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:36.234143 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.234243 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.234473 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":125,"threads_started":1}
I20260810 15:24:36.234807 4353 external_mini_cluster.cc:1428] Setting key b80412c77f300e4023b0eb44efbac01a
I20260810 15:24:36.235311 24331 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37899
I20260810 15:24:36.235422 24331 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:36.235610 24331 heartbeater.cc:511] Master 127.4.64.126:37899 requested a full tablet report, sending...
I20260810 15:24:36.236279 24124 ts_manager.cc:194] Registered new tserver with Master: 57e5101758fd4ec99c9f3ca83be9277a (127.4.64.65:39553)
I20260810 15:24:36.236940 24124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:45215
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:24:36.268334 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37899
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:36.353142 24337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:36.353413 24337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:36.353456 24337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:36.354664 24337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:36.354743 24337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:36.354785 24337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:36.356274 24337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:36.356560 24337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:36.356820 24337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:36.362623 24342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.362617 24345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.362654 24343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:36.363004 24337 server_base.cc:1034] running on GCE node
I20260810 15:24:36.363188 24337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:36.363399 24337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:36.364570 24337 hybrid_clock.cc:648] HybridClock initialized: now 1786375476364561 us; error 39 us; skew 500 ppm
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:36.367297 24337 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:24:36.367727 24337 webserver.cc:460] Webserver started at http://127.4.64.66:45209/ using document root <none> and password file <none>
I20260810 15:24:36.367921 24337 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:36.367964 24337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:36.368062 24337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:36.369107 24337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "ff69e89c4dd5494d80a13e84e2dab0dd"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "ef2116f168210cc8dac3eb1dfe3e4302"
server_key_iv: "c75a3ccb362083f6109ca588bb461bba"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.369426 24337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "ff69e89c4dd5494d80a13e84e2dab0dd"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "ef2116f168210cc8dac3eb1dfe3e4302"
server_key_iv: "c75a3ccb362083f6109ca588bb461bba"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.370807 24337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:36.371534 24352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.371712 24337 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:24:36.371809 24337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "ff69e89c4dd5494d80a13e84e2dab0dd"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "ef2116f168210cc8dac3eb1dfe3e4302"
server_key_iv: "c75a3ccb362083f6109ca588bb461bba"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.371872 24337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:36.381779 24337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:36.383980 24337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:36.384104 24337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:36.384320 24337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:36.384673 24337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:36.384711 24337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.384733 24337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:36.384784 24337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.390862 24337 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:42499
I20260810 15:24:36.391037 24479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:42499 every 8 connection(s)
I20260810 15:24:36.391323 24337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:36.394767 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24337
I20260810 15:24:36.394927 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:24:36.395149 4353 external_mini_cluster.cc:1428] Setting key c50b3cdb420b26e2f0e9c137d4146928
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:36.401916 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.392136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:33973 (local address 127.4.64.126:37899)
0810 15:24:36.392316 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.392318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.392428 (+ 110us) server_negotiation.cc:407] Connection header received
0810 15:24:36.392994 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.392996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.393008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.393063 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:36.394155 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.394696 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.395882 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.396085 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.396160 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.398813 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:36.398814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:36.398814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:36.398821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:36.400274 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.400821 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.400822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.400822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.400876 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.401174 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.401174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.401175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.401287 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:36.401337 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.401746 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.401844 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260810 15:24:36.402249 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.392050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37899 (local address 127.4.64.66:33973)
0810 15:24:36.392281 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.392333 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.392884 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.393135 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.393139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.393822 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:36.394045 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.394051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.394802 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.394805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.395791 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.395794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.396180 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.396181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.396308 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.396876 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:36.396885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:36.398674 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:36.400342 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:36.400350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:36.400353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:36.400710 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:36.400942 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:36.400943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:36.400944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:36.401104 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:36.401356 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:36.401365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:36.401566 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.401735 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.401944 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260810 15:24:36.402908 24480 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37899
I20260810 15:24:36.403026 24480 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:36.403249 24480 heartbeater.cc:511] Master 127.4.64.126:37899 requested a full tablet report, sending...
I20260810 15:24:36.403671 24124 ts_manager.cc:194] Registered new tserver with Master: ff69e89c4dd5494d80a13e84e2dab0dd (127.4.64.66:42499)
I20260810 15:24:36.404089 24124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:33973
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:24:36.430307 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37899
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:36.521412 24486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:36.521719 24486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:36.521791 24486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:36.523020 24486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:36.523068 24486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:36.523095 24486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:36.524636 24486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:36.524914 24486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:36.525194 24486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:36.531396 24497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.531430 24494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.531430 24495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:36.531545 24486 server_base.cc:1034] running on GCE node
I20260810 15:24:36.531873 24486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:36.532156 24486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:36.533396 24486 hybrid_clock.cc:648] HybridClock initialized: now 1786375476533386 us; error 53 us; skew 500 ppm
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:36.536228 24486 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:24:36.536851 24486 webserver.cc:460] Webserver started at http://127.4.64.67:33987/ using document root <none> and password file <none>
I20260810 15:24:36.537079 24486 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:36.537123 24486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:36.537230 24486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:36.538367 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "3e3c3710b582435e9605c4d3830ee42e"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "fc67cb1d864e7e1142e0c15800023c7f"
server_key_iv: "0b893a3622cf59e343b1b7a7f5c3ad69"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.538688 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "3e3c3710b582435e9605c4d3830ee42e"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "fc67cb1d864e7e1142e0c15800023c7f"
server_key_iv: "0b893a3622cf59e343b1b7a7f5c3ad69"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.540096 24486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:24:36.540901 24506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.541121 24486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:36.541206 24486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "3e3c3710b582435e9605c4d3830ee42e"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "fc67cb1d864e7e1142e0c15800023c7f"
server_key_iv: "0b893a3622cf59e343b1b7a7f5c3ad69"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.541266 24486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:36.553690 24486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:36.556259 24486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:36.556396 24486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:36.556715 24486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:36.557041 24486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:36.557077 24486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.557103 24486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:36.557116 24486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.563534 24486 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:33917
I20260810 15:24:36.563589 24634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:33917 every 8 connection(s)
I20260810 15:24:36.563948 24486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:36.566977 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24486
I20260810 15:24:36.567111 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:24:36.567312 4353 external_mini_cluster.cc:1428] Setting key d64de137ac64543b68caeb722a281655
Aug 10 15:24:36 dist-test-slave-9zdj krb5kdc[24077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:36.575100 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.564788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:38999 (local address 127.4.64.126:37899)
0810 15:24:36.564848 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.564850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.565022 (+ 172us) server_negotiation.cc:407] Connection header received
0810 15:24:36.565590 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.565592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.565604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.565754 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:36.566789 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.567593 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.569040 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.569230 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.569291 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.571921 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:36.571923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:36.571923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:36.571932 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:36.573503 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.573978 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.573979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.573980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.574037 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.574290 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.574291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.574291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.574413 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:36.574469 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.574889 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.574995 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:24:36.575350 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.564680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:37899 (local address 127.4.64.67:38999)
0810 15:24:36.564873 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.564918 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.565461 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.565829 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.565832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.566438 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:36.566648 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.566652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.567737 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.567738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.568946 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.568949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.569329 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.569330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.569445 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.570261 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:36.570271 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:36.571816 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:36.573592 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:36.573599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:36.573601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:36.573907 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:36.574102 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:36.574103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:36.574104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:36.574223 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:36.574509 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:36.574516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:36.574688 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.574860 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.575069 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":117,"threads_started":1}
I20260810 15:24:36.575941 24636 heartbeater.cc:347] Connected to a master server at 127.4.64.126:37899
I20260810 15:24:36.576066 24636 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:36.576349 24636 heartbeater.cc:511] Master 127.4.64.126:37899 requested a full tablet report, sending...
I20260810 15:24:36.576798 24124 ts_manager.cc:194] Registered new tserver with Master: 3e3c3710b582435e9605c4d3830ee42e (127.4.64.67:33917)
I20260810 15:24:36.577234 24124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:38999
I20260810 15:24:36.579556 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:24:36.587734 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.581038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.4.64.126:37899)
0810 15:24:36.581118 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.581120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.581234 (+ 114us) server_negotiation.cc:407] Connection header received
0810 15:24:36.581288 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.581289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.581335 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.581399 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:36.582063 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.582613 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.583397 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.583606 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.583688 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.584930 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:36.584931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:36.584931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:36.584939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:36.586606 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.586972 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.586973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.586973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.587034 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:36.587217 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:36.587217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:36.587218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:36.587379 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:36.587459 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.587584 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.587665 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:24:36.589429 24124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:24:36.591143 24124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:24:36.597954 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.593653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:42499 (local address 127.0.0.1:49256)
0810 15:24:36.593972 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.593984 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.594044 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.594630 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.594633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.594636 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:36.594710 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.594712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.595632 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.595633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.596685 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.596687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.597598 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.597599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.597727 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.597742 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.597818 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.597868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":103,"threads_started":1}
I20260810 15:24:36.597991 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.593772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.4.64.66:42499)
0810 15:24:36.594069 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.594074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.594084 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:36.594162 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.594165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.594180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.594297 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:36.594800 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.595526 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.596782 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.597503 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.597587 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.597689 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.597850 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.597896 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260810 15:24:36.598850 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.593914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:39553 (local address 127.0.0.1:50924)
0810 15:24:36.594289 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.594301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.594371 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.594713 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.594713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.594716 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:36.594770 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.594771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.596290 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.596291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.597122 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.597124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.598602 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.598603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.598675 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.598681 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.598728 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.598768 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":116,"threads_started":1}
I20260810 15:24:36.598762 24406 tablet_service.cc:1467] Processing CreateTablet for tablet de0fea908a4940c39bd6b7a7e0b37232 (DEFAULT_TABLE table=test-table [id=ee40401c858240c387c979e49bb0d2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:36.599145 24406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de0fea908a4940c39bd6b7a7e0b37232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:36.599318 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.594268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.4.64.65:39553)
0810 15:24:36.594493 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.594499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.594508 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:24:36.594554 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.594556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.594568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.594708 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:36.594830 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.596101 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.597297 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.598510 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.598594 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.599140 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.599181 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.599227 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":118,"threads_started":1}
I20260810 15:24:36.599814 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.593614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:33917 (local address 127.0.0.1:48224)
0810 15:24:36.593821 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.593866 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.593939 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.594630 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.594634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.594640 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:36.594735 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.594738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.597208 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.597208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.598337 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.598340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.599537 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.599538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.599614 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.599619 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.599700 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.599744 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260810 15:24:36.599850 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.593856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.4.64.67:33917)
0810 15:24:36.594451 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.594456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.594466 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:36.594504 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.594507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.594522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.594607 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:36.594851 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.596102 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.598431 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.599351 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.599433 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.599497 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.599722 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.599765 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":81,"threads_started":1}
I20260810 15:24:36.599996 24258 tablet_service.cc:1467] Processing CreateTablet for tablet de0fea908a4940c39bd6b7a7e0b37232 (DEFAULT_TABLE table=test-table [id=ee40401c858240c387c979e49bb0d2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:36.600346 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de0fea908a4940c39bd6b7a7e0b37232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:36.602833 24655 tablet_bootstrap.cc:492] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Bootstrap starting.
I20260810 15:24:36.603354 24560 tablet_service.cc:1467] Processing CreateTablet for tablet de0fea908a4940c39bd6b7a7e0b37232 (DEFAULT_TABLE table=test-table [id=ee40401c858240c387c979e49bb0d2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:36.603581 24655 tablet_bootstrap.cc:654] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:36.603710 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de0fea908a4940c39bd6b7a7e0b37232. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:36.604136 24655 log.cc:824] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:36.604868 24655 tablet_bootstrap.cc:492] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: No bootstrap required, opened a new log
I20260810 15:24:36.604951 24655 ts_tablet_manager.cc:1397] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:24:36.605784 24656 tablet_bootstrap.cc:492] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: Bootstrap starting.
I20260810 15:24:36.606467 24656 tablet_bootstrap.cc:654] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:36.606971 24659 tablet_bootstrap.cc:492] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Bootstrap starting.
I20260810 15:24:36.606870 24655 raft_consensus.cc:348] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.607054 24655 raft_consensus.cc:374] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:36.607095 24655 raft_consensus.cc:729] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff69e89c4dd5494d80a13e84e2dab0dd, State: Initialized, Role: FOLLOWER
I20260810 15:24:36.607098 24656 log.cc:824] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:36.607245 24655 consensus_queue.cc:260] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.607585 24480 heartbeater.cc:503] Master 127.4.64.126:37899 was elected leader, sending a full tablet report...
I20260810 15:24:36.607592 24655 ts_tablet_manager.cc:1428] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:24:36.607970 24659 tablet_bootstrap.cc:654] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:36.608247 24656 tablet_bootstrap.cc:492] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: No bootstrap required, opened a new log
I20260810 15:24:36.608321 24656 ts_tablet_manager.cc:1397] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:24:36.608608 24659 log.cc:824] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:36.609390 24659 tablet_bootstrap.cc:492] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: No bootstrap required, opened a new log
I20260810 15:24:36.609489 24659 ts_tablet_manager.cc:1397] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:24:36.610153 24656 raft_consensus.cc:348] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.610322 24656 raft_consensus.cc:374] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:36.610368 24656 raft_consensus.cc:729] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e5101758fd4ec99c9f3ca83be9277a, State: Initialized, Role: FOLLOWER
I20260810 15:24:36.610491 24656 consensus_queue.cc:260] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.610775 24656 ts_tablet_manager.cc:1428] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:24:36.610849 24331 heartbeater.cc:503] Master 127.4.64.126:37899 was elected leader, sending a full tablet report...
I20260810 15:24:36.611181 24659 raft_consensus.cc:348] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.611382 24659 raft_consensus.cc:374] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:36.611428 24659 raft_consensus.cc:729] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3c3710b582435e9605c4d3830ee42e, State: Initialized, Role: FOLLOWER
I20260810 15:24:36.611529 24659 consensus_queue.cc:260] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.611842 24659 ts_tablet_manager.cc:1428] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:24:36.611855 24636 heartbeater.cc:503] Master 127.4.64.126:37899 was elected leader, sending a full tablet report...
I20260810 15:24:36.626950 24664 raft_consensus.cc:482] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:36.627095 24664 raft_consensus.cc:504] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.627559 24664 leader_election.cc:288] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57e5101758fd4ec99c9f3ca83be9277a (127.4.64.65:39553), ff69e89c4dd5494d80a13e84e2dab0dd (127.4.64.66:42499)
I20260810 15:24:36.631295 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.627739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:42499 (local address 127.4.64.67:40771)
0810 15:24:36.627804 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.627817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.627872 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.628061 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.628062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.628064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:36.628155 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.628157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.629225 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.629226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.630344 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.630347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.631026 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.631027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.631116 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.631121 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.631175 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.631224 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 15:24:36.631392 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.627864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:40771 (local address 127.4.64.66:42499)
0810 15:24:36.627916 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.627918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.627928 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:36.627959 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.627960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.627972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.628057 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:36.628299 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.629112 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.630450 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.630941 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.631005 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.631015 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.631261 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.631310 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:24:36.631486 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.627799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:39553 (local address 127.4.64.67:42897)
0810 15:24:36.627993 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.628001 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:36.628084 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:36.628250 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:36.628250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:36.628252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:36.628404 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.628407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.629459 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.629460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.630515 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:36.630518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.631123 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.631124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:36.631282 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.631286 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:36.631352 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:24:36.631391 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
I20260810 15:24:36.631515 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.627864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:42897 (local address 127.4.64.65:39553)
0810 15:24:36.627912 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.627914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.628069 (+ 155us) server_negotiation.cc:407] Connection header received
0810 15:24:36.628151 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.628153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.628166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.628232 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:36.628538 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.629358 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.630608 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.631027 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.631102 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.631113 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.631381 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.631434 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:24:36.631776 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0fea908a4940c39bd6b7a7e0b37232" candidate_uuid: "3e3c3710b582435e9605c4d3830ee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" is_pre_election: true
I20260810 15:24:36.631968 24429 raft_consensus.cc:2454] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3c3710b582435e9605c4d3830ee42e in term 0.
I20260810 15:24:36.632043 24279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0fea908a4940c39bd6b7a7e0b37232" candidate_uuid: "3e3c3710b582435e9605c4d3830ee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e5101758fd4ec99c9f3ca83be9277a" is_pre_election: true
I20260810 15:24:36.632256 24279 raft_consensus.cc:2454] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3c3710b582435e9605c4d3830ee42e in term 0.
I20260810 15:24:36.632237 24507 leader_election.cc:302] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e3c3710b582435e9605c4d3830ee42e, ff69e89c4dd5494d80a13e84e2dab0dd; no voters:
I20260810 15:24:36.632396 24664 raft_consensus.cc:2781] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:24:36.632453 24664 raft_consensus.cc:482] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:24:36.632495 24664 raft_consensus.cc:3037] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:36.633306 24664 raft_consensus.cc:504] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.633463 24664 leader_election.cc:288] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [CANDIDATE]: Term 1 election: Requested vote from peers 57e5101758fd4ec99c9f3ca83be9277a (127.4.64.65:39553), ff69e89c4dd5494d80a13e84e2dab0dd (127.4.64.66:42499)
I20260810 15:24:36.633642 24279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0fea908a4940c39bd6b7a7e0b37232" candidate_uuid: "3e3c3710b582435e9605c4d3830ee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57e5101758fd4ec99c9f3ca83be9277a"
I20260810 15:24:36.633658 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de0fea908a4940c39bd6b7a7e0b37232" candidate_uuid: "3e3c3710b582435e9605c4d3830ee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd"
I20260810 15:24:36.633770 24279 raft_consensus.cc:3037] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:36.633766 24429 raft_consensus.cc:3037] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:36.634549 24429 raft_consensus.cc:2454] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3c3710b582435e9605c4d3830ee42e in term 1.
I20260810 15:24:36.634615 24279 raft_consensus.cc:2454] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3c3710b582435e9605c4d3830ee42e in term 1.
I20260810 15:24:36.634786 24511 leader_election.cc:302] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e3c3710b582435e9605c4d3830ee42e, 57e5101758fd4ec99c9f3ca83be9277a; no voters:
I20260810 15:24:36.634874 24664 raft_consensus.cc:2781] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:36.635039 24664 raft_consensus.cc:686] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 1 LEADER]: Becoming Leader. State: Replica: 3e3c3710b582435e9605c4d3830ee42e, State: Running, Role: LEADER
I20260810 15:24:36.635138 24664 consensus_queue.cc:237] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } }
I20260810 15:24:36.636106 24124 catalog_manager.cc:5495] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e3c3710b582435e9605c4d3830ee42e (127.4.64.67). New cstate: current_term: 1 leader_uuid: "3e3c3710b582435e9605c4d3830ee42e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e3c3710b582435e9605c4d3830ee42e" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 33917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 } health_report { overall_health: UNKNOWN } } }
W20260810 15:24:36.641906 24481 tablet.cc:2369] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:36.644105 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.641538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.4.64.67:33917)
0810 15:24:36.641605 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.641607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.641634 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:24:36.641667 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.641668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.641689 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.641757 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:24:36.641926 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.642505 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.643289 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.643516 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.643593 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.643762 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:24:36.643871 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:24:36.643965 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.644008 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.644027 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:24:36.646126 24279 raft_consensus.cc:1261] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 1 FOLLOWER]: Refusing update from remote peer 3e3c3710b582435e9605c4d3830ee42e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:36.646184 24429 raft_consensus.cc:1261] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 1 FOLLOWER]: Refusing update from remote peer 3e3c3710b582435e9605c4d3830ee42e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:36.646515 24666 consensus_queue.cc:1035] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [LEADER]: Connected to new peer: Peer: permanent_uuid: "57e5101758fd4ec99c9f3ca83be9277a" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:36.646713 24664 consensus_queue.cc:1035] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff69e89c4dd5494d80a13e84e2dab0dd" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 42499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:36.649384 24670 mvcc.cc:205] Tried to move back new op lower bound from 7316993952339988480 to 7316993952298381312. Current Snapshot: MvccSnapshot[applied={T|T < 7316993952339988480}]
I20260810 15:24:36.651626 24125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:36.651752 24125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:36.652606 24125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53db8c287f874c1b913ccf4250558cfc: Sending DeleteTablet for 3 replicas of tablet de0fea908a4940c39bd6b7a7e0b37232
I20260810 15:24:36.652917 24560 tablet_service.cc:1514] Processing DeleteTablet for tablet de0fea908a4940c39bd6b7a7e0b37232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:36 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:48224
I20260810 15:24:36.652987 24406 tablet_service.cc:1514] Processing DeleteTablet for tablet de0fea908a4940c39bd6b7a7e0b37232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:36 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:49256
I20260810 15:24:36.653153 24258 tablet_service.cc:1514] Processing DeleteTablet for tablet de0fea908a4940c39bd6b7a7e0b37232 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:36 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:50924
I20260810 15:24:36.653338 24681 tablet_replica.cc:330] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: stopping tablet replica
I20260810 15:24:36.653378 24682 tablet_replica.cc:330] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a: stopping tablet replica
I20260810 15:24:36.653450 24681 raft_consensus.cc:2229] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:36.653543 24681 raft_consensus.cc:2258] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:36.653663 24682 raft_consensus.cc:2229] T de0fea908a4940c39bd6b7a7e0b37232 P 57e5101758fd4ec99c9f3ca83be9277a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:36.653676 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24192
I20260810 15:24:36.662354 24681 ts_tablet_manager.cc:1905] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:36.664567 24681 ts_tablet_manager.cc:1918] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:36.664662 24681 log.cc:1196] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/de0fea908a4940c39bd6b7a7e0b37232
I20260810 15:24:36.664943 24681 ts_tablet_manager.cc:1939] T de0fea908a4940c39bd6b7a7e0b37232 P ff69e89c4dd5494d80a13e84e2dab0dd: Deleting consensus metadata
I20260810 15:24:36.665488 24107 catalog_manager.cc:4841] TS ff69e89c4dd5494d80a13e84e2dab0dd (127.4.64.66:42499): tablet de0fea908a4940c39bd6b7a7e0b37232 (table test-table [id=ee40401c858240c387c979e49bb0d2b7]) successfully deleted
I20260810 15:24:36.665767 24680 tablet_replica.cc:330] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: stopping tablet replica
I20260810 15:24:36.665897 24680 raft_consensus.cc:2229] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:24:36.666006 24680 raft_consensus.cc:2258] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:36.666395 24680 ts_tablet_manager.cc:1905] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:36.668008 24680 ts_tablet_manager.cc:1918] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:36.668094 24680 log.cc:1196] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/de0fea908a4940c39bd6b7a7e0b37232
I20260810 15:24:36.668462 24680 ts_tablet_manager.cc:1939] T de0fea908a4940c39bd6b7a7e0b37232 P 3e3c3710b582435e9605c4d3830ee42e: Deleting consensus metadata
I20260810 15:24:36.668892 24112 catalog_manager.cc:4841] TS 3e3c3710b582435e9605c4d3830ee42e (127.4.64.67:33917): tablet de0fea908a4940c39bd6b7a7e0b37232 (table test-table [id=ee40401c858240c387c979e49bb0d2b7]) successfully deleted
W20260810 15:24:36.669088 24112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:39553 (error 108)
I20260810 15:24:36.669344 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24337
I20260810 15:24:36.671669 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.669525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:39553 (local address 127.0.0.1:50930)
0810 15:24:36.671571 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:36.671624 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:39553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2018}
W20260810 15:24:36.671846 24112 catalog_manager.cc:4584] TS 57e5101758fd4ec99c9f3ca83be9277a (127.4.64.65:39553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de0fea908a4940c39bd6b7a7e0b37232: Network error: Client connection negotiation failed: client connection to 127.4.64.65:39553: connect: Connection refused (error 111)
I20260810 15:24:36.678959 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24486
I20260810 15:24:36.687474 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24091
2026-08-10T15:24:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T15:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:36Z Disabled control of system clock
I20260810 15:24:36.720252 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:39727
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:36.805161 24687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:36.805400 24687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:36.805423 24687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:36.806602 24687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:36.806643 24687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:36.806656 24687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:36.806669 24687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:36.806682 24687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:36.808013 24687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:39727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:36.808228 24687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:36.808481 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:36.813747 24692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.813742 24693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:36.813824 24687 server_base.cc:1034] running on GCE node
W20260810 15:24:36.813742 24695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:36.814313 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:36.814661 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:36.815811 24687 hybrid_clock.cc:648] HybridClock initialized: now 1786375476815796 us; error 52 us; skew 500 ppm
I20260810 15:24:36.817196 24687 webserver.cc:460] Webserver started at http://127.4.64.126:37325/ using document root <none> and password file <none>
I20260810 15:24:36.817512 24687 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:36.817560 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:36.817744 24687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:36.818928 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "b959b888cc6b477fad23728f30e63f76"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "92b44e114b7cb6818e6082be844f3196"
server_key_iv: "8ccc693fa43cca8dd675abd41b386f73"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.819319 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "b959b888cc6b477fad23728f30e63f76"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "92b44e114b7cb6818e6082be844f3196"
server_key_iv: "8ccc693fa43cca8dd675abd41b386f73"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.820751 24687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:36.821586 24702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.821823 24687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:36.821938 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "b959b888cc6b477fad23728f30e63f76"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "92b44e114b7cb6818e6082be844f3196"
server_key_iv: "8ccc693fa43cca8dd675abd41b386f73"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.822052 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:36.833942 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:36.834317 24687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:36.834483 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:36.838675 24687 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:39727
I20260810 15:24:36.838706 24762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:39727 every 8 connection(s)
I20260810 15:24:36.839082 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:36.840149 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:36.842609 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76: Bootstrap starting.
I20260810 15:24:36.843241 24763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:36.843549 24763 log.cc:824] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:36.844345 24763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76: No bootstrap required, opened a new log
I20260810 15:24:36.845480 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24687
I20260810 15:24:36.845558 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:36.845847 4353 external_mini_cluster.cc:1428] Setting key b89e643b61569caba44aa894ae651bbc
I20260810 15:24:36.846206 24763 raft_consensus.cc:348] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b959b888cc6b477fad23728f30e63f76" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39727 } }
I20260810 15:24:36.846451 24763 raft_consensus.cc:374] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:36.846516 24763 raft_consensus.cc:729] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b959b888cc6b477fad23728f30e63f76, State: Initialized, Role: FOLLOWER
I20260810 15:24:36.846683 24763 consensus_queue.cc:260] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b959b888cc6b477fad23728f30e63f76" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39727 } }
I20260810 15:24:36.846776 24763 raft_consensus.cc:388] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:36.846820 24763 raft_consensus.cc:482] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:36.846868 24763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:36.847436 24763 raft_consensus.cc:504] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b959b888cc6b477fad23728f30e63f76" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39727 } }
I20260810 15:24:36.847565 24763 leader_election.cc:302] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b959b888cc6b477fad23728f30e63f76; no voters:
I20260810 15:24:36.847749 24763 leader_election.cc:288] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:36.847802 24769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:36.847980 24769 raft_consensus.cc:686] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [term 1 LEADER]: Becoming Leader. State: Replica: b959b888cc6b477fad23728f30e63f76, State: Running, Role: LEADER
I20260810 15:24:36.848035 24763 sys_catalog.cc:564] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:36.848345 24769 consensus_queue.cc:237] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b959b888cc6b477fad23728f30e63f76" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39727 } }
I20260810 15:24:36.849180 24770 sys_catalog.cc:455] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b959b888cc6b477fad23728f30e63f76. Latest consensus state: current_term: 1 leader_uuid: "b959b888cc6b477fad23728f30e63f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b959b888cc6b477fad23728f30e63f76" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39727 } } }
I20260810 15:24:36.849278 24770 sys_catalog.cc:458] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:36.849638 24782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:36.850004 24771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b959b888cc6b477fad23728f30e63f76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b959b888cc6b477fad23728f30e63f76" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 39727 } } }
I20260810 15:24:36.850103 24771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:36.850193 24782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:36.852031 24782 catalog_manager.cc:1347] Generated new cluster ID: 8b3ee4cb95a94b949d0852fd32365a83
I20260810 15:24:36.852092 24782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:36.855005 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:36.846518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.4.64.126:39727)
0810 15:24:36.846776 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:36.846784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:36.846836 (+ 52us) server_negotiation.cc:407] Connection header received
0810 15:24:36.846958 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:36.846965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:36.847029 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:36.847184 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:36.847506 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.849521 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.850769 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:36.851376 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:36.851531 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:36.854115 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:36.854120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:36.854120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:36.854393 (+ 273us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:36.854428 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:36.854440 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:36.854523 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:36.854620 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 15:24:36.854708 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260810 15:24:36.867144 24782 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:36.867884 24782 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:36.870347 24782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76: Generated new TSK 0
I20260810 15:24:36.870527 24782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:36.875878 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39727
--builtin_ntp_servers=127.4.64.84:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:36.962517 24790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:36.962767 24790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:36.962805 24790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:36.963961 24790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:36.964025 24790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:36.964067 24790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:36.965518 24790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:36.965821 24790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:36.966140 24790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:36.972100 24799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.972131 24797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:36.972105 24796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:36.972704 24790 server_base.cc:1034] running on GCE node
I20260810 15:24:36.972932 24790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:36.973201 24790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:36.974370 24790 hybrid_clock.cc:648] HybridClock initialized: now 1786375476974356 us; error 30 us; skew 500 ppm
I20260810 15:24:36.975965 24790 webserver.cc:460] Webserver started at http://127.4.64.65:34563/ using document root <none> and password file <none>
I20260810 15:24:36.976241 24790 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:36.976301 24790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:36.976437 24790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:36.977922 24790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "5678e312e835497cba0509917c838168"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "5dfb512a59e2c06278cfb0cb93e56697"
server_key_iv: "0c16e8e3b0d273bd589d9b837bf3006e"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.978485 24790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "5678e312e835497cba0509917c838168"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "5dfb512a59e2c06278cfb0cb93e56697"
server_key_iv: "0c16e8e3b0d273bd589d9b837bf3006e"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.980414 24790 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:24:36.981408 24805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.981709 24790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:36.981805 24790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "5678e312e835497cba0509917c838168"
format_stamp: "Formatted at 2026-08-10 15:24:36 on dist-test-slave-9zdj"
server_key: "5dfb512a59e2c06278cfb0cb93e56697"
server_key_iv: "0c16e8e3b0d273bd589d9b837bf3006e"
server_key_version: "encryptionkey@0"
I20260810 15:24:36.981884 24790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:36.992115 24790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:36.992486 24790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:36.992661 24790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:36.992909 24790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:36.993290 24790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:36.993348 24790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.993384 24790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:36.993412 24790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:36.999518 24790 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:35025
I20260810 15:24:36.999567 24931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:35025 every 8 connection(s)
I20260810 15:24:36.999902 24790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:24:37.002403 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24790
I20260810 15:24:37.002519 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:24:37.002725 4353 external_mini_cluster.cc:1428] Setting key 77d17b0073c8ea4852e59ae1b9cf4cbd
I20260810 15:24:37.003809 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39727
--builtin_ntp_servers=127.4.64.84:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:24:37.005812 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.000700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:42375 (local address 127.4.64.126:39727)
0810 15:24:37.000784 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.000787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.000997 (+ 210us) server_negotiation.cc:407] Connection header received
0810 15:24:37.001576 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.001577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.001593 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.001717 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:37.002430 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.003097 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.004203 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.004420 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.004499 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.005457 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:37.005459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:37.005460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:37.005465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:37.005483 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:37.005489 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:37.005522 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.005720 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.005750 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:24:37.006083 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.000653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:39727 (local address 127.4.64.65:42375)
0810 15:24:37.000876 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.000912 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.001462 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.001803 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.001807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.001823 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:37.002323 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.002327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.003199 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.003200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.004107 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.004109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.004508 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.004509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.004659 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.005274 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:37.005275 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:37.005303 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:37.005325 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:37.005568 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:37.005574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:37.005577 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.005658 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.005803 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260810 15:24:37.006661 24932 heartbeater.cc:347] Connected to a master server at 127.4.64.126:39727
I20260810 15:24:37.006784 24932 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:37.006989 24932 heartbeater.cc:511] Master 127.4.64.126:39727 requested a full tablet report, sending...
I20260810 15:24:37.007545 24720 ts_manager.cc:194] Registered new tserver with Master: 5678e312e835497cba0509917c838168 (127.4.64.65:35025)
I20260810 15:24:37.008154 24720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.65:42375
W20260810 15:24:37.092545 24935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:37.092795 24935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:37.092835 24935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:37.094084 24935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:37.094146 24935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:37.094188 24935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:37.095561 24935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:37.095841 24935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:37.096144 24935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:37.101574 24943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:37.101596 24935 server_base.cc:1034] running on GCE node
W20260810 15:24:37.101682 24946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:37.101581 24942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:37.102052 24935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:37.102370 24935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:37.103618 24935 hybrid_clock.cc:648] HybridClock initialized: now 1786375477103570 us; error 39 us; skew 500 ppm
I20260810 15:24:37.105170 24935 webserver.cc:460] Webserver started at http://127.4.64.66:33421/ using document root <none> and password file <none>
I20260810 15:24:37.105435 24935 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:37.105508 24935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:37.105716 24935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:37.106851 24935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "b859b97511be432c8bbf0d33f0d84e04"
format_stamp: "Formatted at 2026-08-10 15:24:37 on dist-test-slave-9zdj"
server_key: "0c93490ecc59cf8a84a2352998a4a0f0"
server_key_iv: "a623f4be623706aa6b971cd39da12bc7"
server_key_version: "encryptionkey@0"
I20260810 15:24:37.107235 24935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "b859b97511be432c8bbf0d33f0d84e04"
format_stamp: "Formatted at 2026-08-10 15:24:37 on dist-test-slave-9zdj"
server_key: "0c93490ecc59cf8a84a2352998a4a0f0"
server_key_iv: "a623f4be623706aa6b971cd39da12bc7"
server_key_version: "encryptionkey@0"
I20260810 15:24:37.108671 24935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:24:37.109543 24952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:37.109767 24935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:37.109895 24935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "b859b97511be432c8bbf0d33f0d84e04"
format_stamp: "Formatted at 2026-08-10 15:24:37 on dist-test-slave-9zdj"
server_key: "0c93490ecc59cf8a84a2352998a4a0f0"
server_key_iv: "a623f4be623706aa6b971cd39da12bc7"
server_key_version: "encryptionkey@0"
I20260810 15:24:37.109965 24935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:37.119771 24935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:37.120117 24935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:37.120237 24935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:37.120438 24935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:37.120784 24935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:37.120824 24935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:37.120846 24935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:37.120858 24935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:37.126910 24935 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:38971
I20260810 15:24:37.127029 25076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:38971 every 8 connection(s)
I20260810 15:24:37.127344 24935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:37.130263 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 24935
I20260810 15:24:37.130349 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:24:37.130591 4353 external_mini_cluster.cc:1428] Setting key 26b96324e673e5a0ae881f03b28e8ada
I20260810 15:24:37.131734 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39727
--builtin_ntp_servers=127.4.64.84:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:24:37.133469 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.128122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:58029 (local address 127.4.64.126:39727)
0810 15:24:37.128252 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.128255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.128534 (+ 279us) server_negotiation.cc:407] Connection header received
0810 15:24:37.129033 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.129034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.129051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.129111 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:37.129813 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.130459 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.131989 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.132182 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.132246 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.133186 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:37.133187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:37.133187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:37.133191 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:37.133228 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:37.133232 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:37.133279 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.133385 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.133410 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:24:37.133803 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.128117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:39727 (local address 127.4.64.66:58029)
0810 15:24:37.128409 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.128447 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.128930 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.129164 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.129167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.129175 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:37.129707 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.129711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.130575 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.130576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.131907 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.131911 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.132277 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.132278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.132456 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.133023 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:37.133024 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:37.133064 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:37.133094 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:37.133299 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:37.133305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:37.133308 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.133379 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.133507 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260810 15:24:37.134369 25077 heartbeater.cc:347] Connected to a master server at 127.4.64.126:39727
I20260810 15:24:37.134485 25077 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:37.134707 25077 heartbeater.cc:511] Master 127.4.64.126:39727 requested a full tablet report, sending...
I20260810 15:24:37.135076 24720 ts_manager.cc:194] Registered new tserver with Master: b859b97511be432c8bbf0d33f0d84e04 (127.4.64.66:38971)
I20260810 15:24:37.135463 24720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.66:58029
W20260810 15:24:37.218191 25081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:37.218422 25081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:37.218444 25081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:37.219616 25081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:37.219663 25081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:37.219688 25081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:37.221041 25081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:37.221307 25081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:37.221647 25081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:37.227106 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:37.227097 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:37.227202 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:37.227540 25081 server_base.cc:1034] running on GCE node
I20260810 15:24:37.227802 25081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:37.228103 25081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:37.229449 25081 hybrid_clock.cc:648] HybridClock initialized: now 1786375477229432 us; error 27 us; skew 500 ppm
I20260810 15:24:37.230861 25081 webserver.cc:460] Webserver started at http://127.4.64.67:36733/ using document root <none> and password file <none>
I20260810 15:24:37.231154 25081 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:37.231334 25081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:37.231509 25081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:37.232746 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "92e2c36df8d345fd838f0627150ee5b4"
format_stamp: "Formatted at 2026-08-10 15:24:37 on dist-test-slave-9zdj"
server_key: "836339495225b7c24393a668dd1dd90b"
server_key_iv: "a87cacf82db587820ba65a4349fab501"
server_key_version: "encryptionkey@0"
I20260810 15:24:37.233186 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "92e2c36df8d345fd838f0627150ee5b4"
format_stamp: "Formatted at 2026-08-10 15:24:37 on dist-test-slave-9zdj"
server_key: "836339495225b7c24393a668dd1dd90b"
server_key_iv: "a87cacf82db587820ba65a4349fab501"
server_key_version: "encryptionkey@0"
I20260810 15:24:37.234835 25081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260810 15:24:37.235724 25096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:37.235934 25081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:37.236045 25081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "92e2c36df8d345fd838f0627150ee5b4"
format_stamp: "Formatted at 2026-08-10 15:24:37 on dist-test-slave-9zdj"
server_key: "836339495225b7c24393a668dd1dd90b"
server_key_iv: "a87cacf82db587820ba65a4349fab501"
server_key_version: "encryptionkey@0"
I20260810 15:24:37.236233 25081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:37.244874 25081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:37.245249 25081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:37.245412 25081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:37.245755 25081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:37.246114 25081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:37.246181 25081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:37.246224 25081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:37.246251 25081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:37.252791 25081 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:37573
I20260810 15:24:37.252848 25215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:37573 every 8 connection(s)
I20260810 15:24:37.253270 25081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:37.258857 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25081
I20260810 15:24:37.259007 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:24:37.259277 4353 external_mini_cluster.cc:1428] Setting key a9491363780f9de869b98c42f737f321
I20260810 15:24:37.259939 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.254469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:48871 (local address 127.4.64.126:39727)
0810 15:24:37.254580 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.254583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.254827 (+ 244us) server_negotiation.cc:407] Connection header received
0810 15:24:37.255435 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.255437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.255452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.255504 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:37.256311 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.257030 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.258217 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.258464 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.258572 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.259606 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:37.259608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:37.259608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:37.259615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:37.259632 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:37.259637 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:37.259692 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.259814 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.259852 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 15:24:37.260353 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.254371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:39727 (local address 127.4.64.67:48871)
0810 15:24:37.254676 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.254716 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.255308 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.255577 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.255590 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.255599 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:37.256179 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.256186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.257148 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.257149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.258101 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.258104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.258586 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.258587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.258755 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.259435 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:37.259436 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:37.259468 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:37.259494 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:37.259705 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:37.259711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:37.259715 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.259814 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.259936 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
I20260810 15:24:37.260885 25216 heartbeater.cc:347] Connected to a master server at 127.4.64.126:39727
I20260810 15:24:37.260982 25216 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:37.261215 25216 heartbeater.cc:511] Master 127.4.64.126:39727 requested a full tablet report, sending...
I20260810 15:24:37.261569 24720 ts_manager.cc:194] Registered new tserver with Master: 92e2c36df8d345fd838f0627150ee5b4 (127.4.64.67:37573)
I20260810 15:24:37.262007 24720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.67:48871
I20260810 15:24:37.271449 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:24:37.276062 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.272987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.4.64.126:39727)
0810 15:24:37.273062 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.273064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.273153 (+ 89us) server_negotiation.cc:407] Connection header received
0810 15:24:37.273222 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.273223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.273252 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.273303 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:37.273515 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.274055 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.274877 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.275064 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.275120 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.275832 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:37.275833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:37.275833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:37.275837 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:37.275851 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:37.275854 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:37.275899 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.275936 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.275958 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:24:37.277895 24720 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:24:37.279558 24720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:24:37.287045 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.282064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37573 (local address 127.0.0.1:40104)
0810 15:24:37.282577 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.282609 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.282663 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.283125 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.283125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.283127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:37.283176 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.283177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.284236 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.284237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.285800 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.285803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.286688 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.286689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.286785 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.286802 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.286874 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.286933 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":57,"threads_started":1}
I20260810 15:24:37.287300 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.282380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.4.64.65:35025)
0810 15:24:37.282689 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.282694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.282707 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:24:37.283127 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.283131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.283148 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.283253 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:37.283472 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.284526 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.285817 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.286603 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.286667 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.287119 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.287148 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.287197 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260810 15:24:37.287429 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.282163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.4.64.67:37573)
0810 15:24:37.282977 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.282981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.282991 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:37.283031 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.283033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.283049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.283127 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:37.283263 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.284133 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.285897 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.286592 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.286669 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.286764 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.287283 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.287330 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":99,"threads_started":1}
I20260810 15:24:37.287504 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.282378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.4.64.66:38971)
0810 15:24:37.282584 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.282590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.282857 (+ 267us) server_negotiation.cc:407] Connection header received
0810 15:24:37.282913 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.282915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.282933 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.283015 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:37.283230 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.284377 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.285932 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.286792 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.286886 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.286972 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.287029 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.287070 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":123,"threads_started":1}
I20260810 15:24:37.287593 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.282094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38971 (local address 127.0.0.1:43784)
0810 15:24:37.282783 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.282793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.282840 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.283039 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.283043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.283046 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:37.283116 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.283118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.284468 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.284469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.285836 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.285838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.286898 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.286899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.286950 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.286954 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.287013 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.287048 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":29,"spinlock_wait_cycles":384,"thread_start_us":125,"threads_started":1}
I20260810 15:24:37.287992 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.282094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:35025 (local address 127.0.0.1:33718)
0810 15:24:37.282607 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.282621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.282780 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.283339 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.283340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.283342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:37.283379 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.283380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.284606 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.284607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.285732 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.285735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.286750 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.286751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.286820 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.286825 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.286894 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.286935 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":204,"threads_started":1}
I20260810 15:24:37.288157 25147 tablet_service.cc:1467] Processing CreateTablet for tablet 01c33710fcc346719b225cca8e5c0a3d (DEFAULT_TABLE table=test-table [id=c97bc8dc6b1a4b259f6d0c2f676d5b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:37.288285 25005 tablet_service.cc:1467] Processing CreateTablet for tablet 01c33710fcc346719b225cca8e5c0a3d (DEFAULT_TABLE table=test-table [id=c97bc8dc6b1a4b259f6d0c2f676d5b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:37.288576 25147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c33710fcc346719b225cca8e5c0a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:37.288702 25005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c33710fcc346719b225cca8e5c0a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:37.288705 24859 tablet_service.cc:1467] Processing CreateTablet for tablet 01c33710fcc346719b225cca8e5c0a3d (DEFAULT_TABLE table=test-table [id=c97bc8dc6b1a4b259f6d0c2f676d5b3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:37.289032 24859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c33710fcc346719b225cca8e5c0a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:37.291576 25231 tablet_bootstrap.cc:492] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Bootstrap starting.
I20260810 15:24:37.292235 25232 tablet_bootstrap.cc:492] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Bootstrap starting.
I20260810 15:24:37.292466 25233 tablet_bootstrap.cc:492] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168: Bootstrap starting.
I20260810 15:24:37.292376 25231 tablet_bootstrap.cc:654] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:37.292920 25232 tablet_bootstrap.cc:654] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:37.292903 25231 log.cc:824] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:37.293087 25233 tablet_bootstrap.cc:654] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:37.293253 25232 log.cc:824] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:37.293426 25233 log.cc:824] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:37.294124 25232 tablet_bootstrap.cc:492] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: No bootstrap required, opened a new log
I20260810 15:24:37.294190 25233 tablet_bootstrap.cc:492] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168: No bootstrap required, opened a new log
I20260810 15:24:37.294220 25232 ts_tablet_manager.cc:1397] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:24:37.294289 25233 ts_tablet_manager.cc:1397] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:24:37.295984 25232 raft_consensus.cc:348] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.296038 25233 raft_consensus.cc:348] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.296146 25232 raft_consensus.cc:374] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:37.296161 25233 raft_consensus.cc:374] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:37.296203 25232 raft_consensus.cc:729] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b859b97511be432c8bbf0d33f0d84e04, State: Initialized, Role: FOLLOWER
I20260810 15:24:37.296236 25233 raft_consensus.cc:729] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5678e312e835497cba0509917c838168, State: Initialized, Role: FOLLOWER
I20260810 15:24:37.296322 25232 consensus_queue.cc:260] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.296347 25233 consensus_queue.cc:260] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.296602 25233 ts_tablet_manager.cc:1428] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 15:24:37.296598 25232 ts_tablet_manager.cc:1428] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:24:37.296625 25077 heartbeater.cc:503] Master 127.4.64.126:39727 was elected leader, sending a full tablet report...
I20260810 15:24:37.296741 24932 heartbeater.cc:503] Master 127.4.64.126:39727 was elected leader, sending a full tablet report...
I20260810 15:24:37.296953 25231 tablet_bootstrap.cc:492] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: No bootstrap required, opened a new log
I20260810 15:24:37.297024 25231 ts_tablet_manager.cc:1397] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 15:24:37.298713 25239 raft_consensus.cc:482] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:37.298794 25239 raft_consensus.cc:504] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.299090 25239 leader_election.cc:288] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b859b97511be432c8bbf0d33f0d84e04 (127.4.64.66:38971), 92e2c36df8d345fd838f0627150ee5b4 (127.4.64.67:37573)
I20260810 15:24:37.299671 25231 raft_consensus.cc:348] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.299832 25231 raft_consensus.cc:374] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:37.299875 25231 raft_consensus.cc:729] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e2c36df8d345fd838f0627150ee5b4, State: Initialized, Role: FOLLOWER
I20260810 15:24:37.299989 25231 consensus_queue.cc:260] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.300292 25231 ts_tablet_manager.cc:1428] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:24:37.300292 25216 heartbeater.cc:503] Master 127.4.64.126:39727 was elected leader, sending a full tablet report...
I20260810 15:24:37.302736 25229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.299259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:35395 (local address 127.4.64.66:38971)
0810 15:24:37.299303 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.299305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.299315 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:37.299369 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.299370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.299382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.299447 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:37.299712 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.300562 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.301891 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.302429 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.302489 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.302498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.302653 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.302683 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:24:37.302790 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.299211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:38971 (local address 127.4.64.65:35395)
0810 15:24:37.299263 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.299275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.299312 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.299503 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.299505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.299506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:37.299574 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.299576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.300646 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.300647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.301810 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.301813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.302508 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.302509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.302579 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.302583 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.302648 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.302691 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:24:37.303110 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.299360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37573 (local address 127.4.64.65:50675)
0810 15:24:37.299515 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.299525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:37.299568 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:37.299943 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:37.299944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:37.299946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:37.300015 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.300017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.300925 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.300926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.302131 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:37.302133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.302868 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.302869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:37.302943 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.302945 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:24:37.302996 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:24:37.303032 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260810 15:24:37.303145 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.299752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:50675 (local address 127.4.64.67:37573)
0810 15:24:37.299796 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.299798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.299806 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:24:37.299829 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.299830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.299839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.299895 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:37.300095 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.300818 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.302276 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.302752 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.302815 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.302826 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.303025 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.303083 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:24:37.303154 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c33710fcc346719b225cca8e5c0a3d" candidate_uuid: "5678e312e835497cba0509917c838168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b859b97511be432c8bbf0d33f0d84e04" is_pre_election: true
I20260810 15:24:37.303329 25026 raft_consensus.cc:2454] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5678e312e835497cba0509917c838168 in term 0.
I20260810 15:24:37.303536 24807 leader_election.cc:302] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5678e312e835497cba0509917c838168, b859b97511be432c8bbf0d33f0d84e04; no voters:
I20260810 15:24:37.303598 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c33710fcc346719b225cca8e5c0a3d" candidate_uuid: "5678e312e835497cba0509917c838168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e2c36df8d345fd838f0627150ee5b4" is_pre_election: true
I20260810 15:24:37.303659 25239 raft_consensus.cc:2781] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:24:37.303720 25239 raft_consensus.cc:482] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:24:37.303753 25239 raft_consensus.cc:3037] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:37.303740 25167 raft_consensus.cc:2454] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5678e312e835497cba0509917c838168 in term 0.
I20260810 15:24:37.304492 25239 raft_consensus.cc:504] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.304642 25239 leader_election.cc:288] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [CANDIDATE]: Term 1 election: Requested vote from peers b859b97511be432c8bbf0d33f0d84e04 (127.4.64.66:38971), 92e2c36df8d345fd838f0627150ee5b4 (127.4.64.67:37573)
I20260810 15:24:37.304808 25167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c33710fcc346719b225cca8e5c0a3d" candidate_uuid: "5678e312e835497cba0509917c838168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e2c36df8d345fd838f0627150ee5b4"
I20260810 15:24:37.304800 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c33710fcc346719b225cca8e5c0a3d" candidate_uuid: "5678e312e835497cba0509917c838168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b859b97511be432c8bbf0d33f0d84e04"
I20260810 15:24:37.304885 25167 raft_consensus.cc:3037] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:37.304903 25026 raft_consensus.cc:3037] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:37.305753 25167 raft_consensus.cc:2454] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5678e312e835497cba0509917c838168 in term 1.
I20260810 15:24:37.305783 25026 raft_consensus.cc:2454] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5678e312e835497cba0509917c838168 in term 1.
I20260810 15:24:37.305954 24807 leader_election.cc:302] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5678e312e835497cba0509917c838168, b859b97511be432c8bbf0d33f0d84e04; no voters:
I20260810 15:24:37.306061 25239 raft_consensus.cc:2781] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:37.306231 25239 raft_consensus.cc:686] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [term 1 LEADER]: Becoming Leader. State: Replica: 5678e312e835497cba0509917c838168, State: Running, Role: LEADER
I20260810 15:24:37.306335 25239 consensus_queue.cc:237] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } }
I20260810 15:24:37.307228 24719 catalog_manager.cc:5495] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5678e312e835497cba0509917c838168 (127.4.64.65). New cstate: current_term: 1 leader_uuid: "5678e312e835497cba0509917c838168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5678e312e835497cba0509917c838168" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 35025 } health_report { overall_health: HEALTHY } } }
I20260810 15:24:37.314996 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.311797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.4.64.65:35025)
0810 15:24:37.311849 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:37.311851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:37.311859 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:24:37.311938 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:37.311939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:37.311957 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:37.312050 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:24:37.312162 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.312909 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.314059 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:37.314327 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:37.314391 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:37.314551 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:24:37.314660 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:24:37.314725 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:37.314879 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:24:37.314902 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:24:37.317206 25026 raft_consensus.cc:1261] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 1 FOLLOWER]: Refusing update from remote peer 5678e312e835497cba0509917c838168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:37.317584 25167 raft_consensus.cc:1261] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 1 FOLLOWER]: Refusing update from remote peer 5678e312e835497cba0509917c838168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:37.317668 25239 consensus_queue.cc:1035] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b859b97511be432c8bbf0d33f0d84e04" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:37.317911 25242 consensus_queue.cc:1035] T 01c33710fcc346719b225cca8e5c0a3d P 5678e312e835497cba0509917c838168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e2c36df8d345fd838f0627150ee5b4" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:37.320799 25247 mvcc.cc:205] Tried to move back new op lower bound from 7316993955087724544 to 7316993955047505920. Current Snapshot: MvccSnapshot[applied={T|T < 7316993955087724544}]
I20260810 15:24:37.320902 25246 mvcc.cc:205] Tried to move back new op lower bound from 7316993955087724544 to 7316993955047505920. Current Snapshot: MvccSnapshot[applied={T|T < 7316993955087724544}]
I20260810 15:24:37.322973 24719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:37.323081 24719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:37.323885 24719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b959b888cc6b477fad23728f30e63f76: Sending DeleteTablet for 3 replicas of tablet 01c33710fcc346719b225cca8e5c0a3d
I20260810 15:24:37.324236 25005 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c33710fcc346719b225cca8e5c0a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:37 UTC) from {username='slave'} at 127.0.0.1:43784
I20260810 15:24:37.324254 25147 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c33710fcc346719b225cca8e5c0a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:37 UTC) from {username='slave'} at 127.0.0.1:40104
I20260810 15:24:37.324239 24859 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c33710fcc346719b225cca8e5c0a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:37 UTC) from {username='slave'} at 127.0.0.1:33718
I20260810 15:24:37.324627 25254 tablet_replica.cc:330] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: stopping tablet replica
I20260810 15:24:37.324759 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24790
I20260810 15:24:37.327903 25254 raft_consensus.cc:2229] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:37.328032 25254 raft_consensus.cc:2258] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:37.328351 25255 tablet_replica.cc:330] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: stopping tablet replica
I20260810 15:24:37.328454 25255 raft_consensus.cc:2229] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:37.328466 25254 ts_tablet_manager.cc:1905] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:37.328545 25255 raft_consensus.cc:2258] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:37.329969 25255 ts_tablet_manager.cc:1905] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:37.330124 25254 ts_tablet_manager.cc:1918] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:37.330214 25254 log.cc:1196] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/01c33710fcc346719b225cca8e5c0a3d
I20260810 15:24:37.330490 25254 ts_tablet_manager.cc:1939] T 01c33710fcc346719b225cca8e5c0a3d P 92e2c36df8d345fd838f0627150ee5b4: Deleting consensus metadata
I20260810 15:24:37.330945 24705 catalog_manager.cc:4841] TS 92e2c36df8d345fd838f0627150ee5b4 (127.4.64.67:37573): tablet 01c33710fcc346719b225cca8e5c0a3d (table test-table [id=c97bc8dc6b1a4b259f6d0c2f676d5b3e]) successfully deleted
I20260810 15:24:37.331796 25255 ts_tablet_manager.cc:1918] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:37.331858 25255 log.cc:1196] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/01c33710fcc346719b225cca8e5c0a3d
I20260810 15:24:37.332123 25255 ts_tablet_manager.cc:1939] T 01c33710fcc346719b225cca8e5c0a3d P b859b97511be432c8bbf0d33f0d84e04: Deleting consensus metadata
I20260810 15:24:37.332558 24704 catalog_manager.cc:4841] TS b859b97511be432c8bbf0d33f0d84e04 (127.4.64.66:38971): tablet 01c33710fcc346719b225cca8e5c0a3d (table test-table [id=c97bc8dc6b1a4b259f6d0c2f676d5b3e]) successfully deleted
W20260810 15:24:37.334478 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:35025 (error 108)
I20260810 15:24:37.334693 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24935
I20260810 15:24:37.335903 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:37.334901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:35025 (local address 127.0.0.1:33726)
0810 15:24:37.335807 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:37.335861 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:35025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":878}
W20260810 15:24:37.337667 24706 catalog_manager.cc:4584] TS 5678e312e835497cba0509917c838168 (127.4.64.65:35025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01c33710fcc346719b225cca8e5c0a3d: Network error: Client connection negotiation failed: client connection to 127.4.64.65:35025: connect: Connection refused (error 111)
I20260810 15:24:37.341570 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25081
I20260810 15:24:37.350065 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 24687
2026-08-10T15:24:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:37 dist-test-slave-9zdj krb5kdc[25260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:37 dist-test-slave-9zdj krb5kdc[25260](info): set up 2 sockets
Aug 10 15:24:37 dist-test-slave-9zdj krb5kdc[25260](info): commencing operation
krb5kdc: starting...
W20260810 15:24:38.487665 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:38 dist-test-slave-9zdj krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:24:38.630170 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:32883
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:38.727248 25273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:38.727504 25273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:38.727541 25273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:38.728719 25273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:38.728788 25273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:38.728813 25273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:38.728845 25273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:38.728878 25273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:38.730504 25273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:32883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:38.730868 25273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:38.731231 25273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:24:38.736511 25273 server_base.cc:1034] running on GCE node
W20260810 15:24:38.736482 25281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:38.736491 25278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:38.736482 25279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:38.737115 25273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:38.737422 25273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:38.738593 25273 hybrid_clock.cc:648] HybridClock initialized: now 1786375478738564 us; error 47 us; skew 500 ppm
Aug 10 15:24:38 dist-test-slave-9zdj krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:38.741392 25273 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:24:38.741945 25273 webserver.cc:460] Webserver started at http://127.4.64.126:45545/ using document root <none> and password file <none>
I20260810 15:24:38.742220 25273 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:38.742302 25273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:38.742461 25273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:38.743559 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "49fb3b3e88b74c2e9d3fe6186c28daac"
format_stamp: "Formatted at 2026-08-10 15:24:38 on dist-test-slave-9zdj"
server_key: "2baef6e6dc1207407d0adcc0f496a9c0"
server_key_iv: "5c0e1e83cde25935e4cd9ee44fba3f78"
server_key_version: "encryptionkey@0"
I20260810 15:24:38.743940 25273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "49fb3b3e88b74c2e9d3fe6186c28daac"
format_stamp: "Formatted at 2026-08-10 15:24:38 on dist-test-slave-9zdj"
server_key: "2baef6e6dc1207407d0adcc0f496a9c0"
server_key_iv: "5c0e1e83cde25935e4cd9ee44fba3f78"
server_key_version: "encryptionkey@0"
I20260810 15:24:38.745484 25273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:24:38.746511 25289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:38.746838 25273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:38.746954 25273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "49fb3b3e88b74c2e9d3fe6186c28daac"
format_stamp: "Formatted at 2026-08-10 15:24:38 on dist-test-slave-9zdj"
server_key: "2baef6e6dc1207407d0adcc0f496a9c0"
server_key_iv: "5c0e1e83cde25935e4cd9ee44fba3f78"
server_key_version: "encryptionkey@0"
I20260810 15:24:38.747078 25273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:38.770483 25273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:38.773236 25273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:38.773432 25273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:38.777912 25273 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:32883
I20260810 15:24:38.777954 25351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:32883 every 8 connection(s)
I20260810 15:24:38.778373 25273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:38.779389 25352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:38.781742 25352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac: Bootstrap starting.
I20260810 15:24:38.782347 25352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:38.782655 25352 log.cc:824] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:38.783485 25352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac: No bootstrap required, opened a new log
I20260810 15:24:38.785089 25352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32883 } }
I20260810 15:24:38.785251 25352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:38.785276 25352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fb3b3e88b74c2e9d3fe6186c28daac, State: Initialized, Role: FOLLOWER
I20260810 15:24:38.785444 25352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32883 } }
I20260810 15:24:38.785511 25352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:38.785532 25352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:38.785588 25352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:38.786180 25352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32883 } }
I20260810 15:24:38.786273 25352 leader_election.cc:302] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49fb3b3e88b74c2e9d3fe6186c28daac; no voters:
I20260810 15:24:38.786466 25352 leader_election.cc:288] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:38.786525 25355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:38.786753 25352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:38.786842 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25273
I20260810 15:24:38.786875 25355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [term 1 LEADER]: Becoming Leader. State: Replica: 49fb3b3e88b74c2e9d3fe6186c28daac, State: Running, Role: LEADER
I20260810 15:24:38.786981 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:38.786986 25355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32883 } }
I20260810 15:24:38.787297 4353 external_mini_cluster.cc:1428] Setting key 0184dcccf6382d6a5720f6eadebc83ea
I20260810 15:24:38.787472 25356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32883 } } }
I20260810 15:24:38.787482 25357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49fb3b3e88b74c2e9d3fe6186c28daac. Latest consensus state: current_term: 1 leader_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49fb3b3e88b74c2e9d3fe6186c28daac" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 32883 } } }
I20260810 15:24:38.787591 25356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:38.787611 25357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:38.787847 25365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:38.788534 25365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:38.790457 25365 catalog_manager.cc:1347] Generated new cluster ID: 2aa1754ff96847a3ba23cbf792d5cb41
I20260810 15:24:38.790516 25365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:24:38 dist-test-slave-9zdj krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:38.800733 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:38.788056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.4.64.126:32883)
0810 15:24:38.788498 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:38.788508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:38.788543 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:24:38.788676 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:38.788759 (+ 83us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:38.788855 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:38.789014 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:38.789839 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:38.791420 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:38.792557 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:38.793278 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:38.793385 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:38.796483 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:38.796488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:38.796488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:38.796501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:38.798669 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:38.799108 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:38.799112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:38.799113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:38.799306 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:38.799571 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:38.799573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:38.799573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:38.799977 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:38.800177 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:38.800293 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:24:38.800469 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":80,"threads_started":1}
I20260810 15:24:38.806963 25365 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:38.807855 25365 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:38.815037 25365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac: Generated new TSK 0
I20260810 15:24:38.815284 25365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:24:38.850212 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32883
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:38.944770 25379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:38.945019 25379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:38.945047 25379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:38.946516 25379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:38.946565 25379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:38.946595 25379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:38.948341 25379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:38.948628 25379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:38.948899 25379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:38.955348 25389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:38.955318 25387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:38.955468 25379 server_base.cc:1034] running on GCE node
W20260810 15:24:38.955330 25386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:38.955780 25379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:38.956075 25379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:38.957224 25379 hybrid_clock.cc:648] HybridClock initialized: now 1786375478957198 us; error 40 us; skew 500 ppm
Aug 10 15:24:38 dist-test-slave-9zdj krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:38.960234 25379 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:24:38.960774 25379 webserver.cc:460] Webserver started at http://127.4.64.65:37651/ using document root <none> and password file <none>
I20260810 15:24:38.961035 25379 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:38.961107 25379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:38.961267 25379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:38.962460 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "bc94074ceb754433919a91d036233084"
format_stamp: "Formatted at 2026-08-10 15:24:38 on dist-test-slave-9zdj"
server_key: "62c0ff3f4b03393a2550e1ad8c3dc809"
server_key_iv: "0943518025eb98f82c41aee38f88cb36"
server_key_version: "encryptionkey@0"
I20260810 15:24:38.962848 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "bc94074ceb754433919a91d036233084"
format_stamp: "Formatted at 2026-08-10 15:24:38 on dist-test-slave-9zdj"
server_key: "62c0ff3f4b03393a2550e1ad8c3dc809"
server_key_iv: "0943518025eb98f82c41aee38f88cb36"
server_key_version: "encryptionkey@0"
I20260810 15:24:38.964398 25379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:24:38.965358 25399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:38.965703 25379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:38.965839 25379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "bc94074ceb754433919a91d036233084"
format_stamp: "Formatted at 2026-08-10 15:24:38 on dist-test-slave-9zdj"
server_key: "62c0ff3f4b03393a2550e1ad8c3dc809"
server_key_iv: "0943518025eb98f82c41aee38f88cb36"
server_key_version: "encryptionkey@0"
I20260810 15:24:38.965929 25379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:38.984598 25379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:38.987272 25379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:38.987460 25379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:38.987828 25379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:38.988200 25379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:38.988257 25379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:38.988291 25379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:38.988323 25379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:38.994812 25379 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:32927
I20260810 15:24:38.994880 25522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:32927 every 8 connection(s)
I20260810 15:24:38.995256 25379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:24:38.997148 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25379
I20260810 15:24:38.997248 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:24:38.997483 4353 external_mini_cluster.cc:1428] Setting key 48ead515612913100f7acb87a617e223
Aug 10 15:24:38 dist-test-slave-9zdj krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:39.006232 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:38.996111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:47921 (local address 127.4.64.126:32883)
0810 15:24:38.996171 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:38.996173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:38.996432 (+ 259us) server_negotiation.cc:407] Connection header received
0810 15:24:38.996981 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:38.996983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:38.996998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:38.997063 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:38.998409 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:38.998930 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:38.999954 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.000179 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.000275 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.002656 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:39.002658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:39.002658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:39.002666 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:39.004488 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.004916 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.004918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.004918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.004999 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.005398 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.005399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.005399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.005581 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:39.005729 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.006005 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.006114 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:24:39.006255 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:38.996094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32883 (local address 127.4.64.65:47921)
0810 15:24:38.996299 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:38.996337 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:38.996871 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:38.997138 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:38.997142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:38.998045 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:38.998307 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:38.998312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:38.999045 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:38.999046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:38.999858 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:38.999860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.000286 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.000287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.000421 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.000987 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:39.000996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:39.002544 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:39.004585 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:39.004592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:39.004594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:39.004827 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:39.005110 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:39.005111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:39.005111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:39.005306 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:39.005703 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:39.005713 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:39.005852 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.005945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.006074 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260810 15:24:39.006868 25523 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32883
I20260810 15:24:39.007004 25523 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:39.007237 25523 heartbeater.cc:511] Master 127.4.64.126:32883 requested a full tablet report, sending...
I20260810 15:24:39.007822 25309 ts_manager.cc:194] Registered new tserver with Master: bc94074ceb754433919a91d036233084 (127.4.64.65:32927)
I20260810 15:24:39.008450 25309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:47921
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:24:39.029837 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32883
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:39.123262 25528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:39.123502 25528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:39.123528 25528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:39.124761 25528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:39.124805 25528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:39.124835 25528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:39.126435 25528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:39.126684 25528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:39.126889 25528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:39.132754 25533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.132762 25536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.132781 25534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:39.133157 25528 server_base.cc:1034] running on GCE node
I20260810 15:24:39.133409 25528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:39.133716 25528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:39.134891 25528 hybrid_clock.cc:648] HybridClock initialized: now 1786375479134882 us; error 38 us; skew 500 ppm
Aug 10 15:24:39 dist-test-slave-9zdj krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:39.137923 25528 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:24:39.138429 25528 webserver.cc:460] Webserver started at http://127.4.64.66:41261/ using document root <none> and password file <none>
I20260810 15:24:39.138688 25528 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:39.138795 25528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:39.138963 25528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:39.140156 25528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "7c0f5e86958c4dcb9066c680bd0fbc52"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "df7040f13c439eea549ef2cb58c4f2b2"
server_key_iv: "7a494608c395f1dd9b6c4fcdee5577a5"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.140599 25528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "7c0f5e86958c4dcb9066c680bd0fbc52"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "df7040f13c439eea549ef2cb58c4f2b2"
server_key_iv: "7a494608c395f1dd9b6c4fcdee5577a5"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.142271 25528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:39.143262 25543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.143497 25528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:39.143608 25528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "7c0f5e86958c4dcb9066c680bd0fbc52"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "df7040f13c439eea549ef2cb58c4f2b2"
server_key_iv: "7a494608c395f1dd9b6c4fcdee5577a5"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.143711 25528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:39.152592 25528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:39.155055 25528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:39.155303 25528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:39.155584 25528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:39.155965 25528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:39.156034 25528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.156080 25528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:39.156111 25528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.162678 25528 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:36713
I20260810 15:24:39.162744 25672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:36713 every 8 connection(s)
I20260810 15:24:39.163084 25528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:39.166218 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25528
I20260810 15:24:39.166345 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:24:39.166620 4353 external_mini_cluster.cc:1428] Setting key f55a6adb1669b4c07eb4d8e172eed898
Aug 10 15:24:39 dist-test-slave-9zdj krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:39.174242 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.163900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:58609 (local address 127.4.64.126:32883)
0810 15:24:39.164000 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.164003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.164179 (+ 176us) server_negotiation.cc:407] Connection header received
0810 15:24:39.164703 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.164704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.164716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.164774 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:39.165825 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.166438 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.167958 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.168196 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.168252 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.170967 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:39.170969 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:39.170969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:39.170976 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:39.172445 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.173024 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.173025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.173025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.173080 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.173423 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.173424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.173424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.173562 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:39.173666 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.173999 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.174101 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 15:24:39.174320 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.163834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32883 (local address 127.4.64.66:58609)
0810 15:24:39.164042 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.164082 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.164593 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.164829 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.164833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.165490 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:39.165733 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.165738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.166536 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.166537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.167874 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.167877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.168288 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.168290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.168458 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.169018 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:39.169030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:39.170877 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:39.172538 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:39.172546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:39.172549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:39.172943 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:39.173179 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:39.173179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:39.173180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:39.173345 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:39.173679 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:39.173687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:39.173848 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.173947 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.174088 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":135,"threads_started":1}
I20260810 15:24:39.174918 25673 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32883
I20260810 15:24:39.175043 25673 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:39.175246 25673 heartbeater.cc:511] Master 127.4.64.126:32883 requested a full tablet report, sending...
I20260810 15:24:39.175678 25309 ts_manager.cc:194] Registered new tserver with Master: 7c0f5e86958c4dcb9066c680bd0fbc52 (127.4.64.66:36713)
I20260810 15:24:39.176204 25309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:58609
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:24:39.202003 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32883
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:39.291090 25678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:39.291349 25678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:39.291391 25678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:39.292567 25678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:39.292635 25678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:39.292680 25678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:39.294270 25678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:32883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:39.294555 25678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:39.294816 25678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:24:39.300483 25678 server_base.cc:1034] running on GCE node
W20260810 15:24:39.300563 25684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.300655 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.300595 25683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:39.301210 25678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:39.301513 25678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:39.302685 25678 hybrid_clock.cc:648] HybridClock initialized: now 1786375479302673 us; error 51 us; skew 500 ppm
Aug 10 15:24:39 dist-test-slave-9zdj krb5kdc[25260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:39.305495 25678 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:24:39.306056 25678 webserver.cc:460] Webserver started at http://127.4.64.67:41109/ using document root <none> and password file <none>
I20260810 15:24:39.306334 25678 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:39.306384 25678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:39.306551 25678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:39.307651 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "d9be2965b7cf4226a862822d1c417fbe"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "4fdb4913bf9d7f4383a0d9a438a8cc34"
server_key_iv: "fecf4871e32a84175c0f81e3b77d12a6"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.308018 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "d9be2965b7cf4226a862822d1c417fbe"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "4fdb4913bf9d7f4383a0d9a438a8cc34"
server_key_iv: "fecf4871e32a84175c0f81e3b77d12a6"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.309484 25678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:24:39.310395 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.310601 25678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:39.310727 25678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "d9be2965b7cf4226a862822d1c417fbe"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "4fdb4913bf9d7f4383a0d9a438a8cc34"
server_key_iv: "fecf4871e32a84175c0f81e3b77d12a6"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.310791 25678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:39.324054 25678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:39.326869 25678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:39.327018 25678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:39.327314 25678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:39.327687 25678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:39.327751 25678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.327821 25678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:39.327852 25678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.334115 25678 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:37883
I20260810 15:24:39.334201 25820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:37883 every 8 connection(s)
I20260810 15:24:39.334578 25678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:39.338933 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25678
I20260810 15:24:39.339076 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:24:39.339318 4353 external_mini_cluster.cc:1428] Setting key 65f1633995b75569a98af38e1282e61e
Aug 10 15:24:39 dist-test-slave-9zdj krb5kdc[25260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375479, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:39.344923 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.335409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:45233 (local address 127.4.64.126:32883)
0810 15:24:39.335498 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.335501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.335693 (+ 192us) server_negotiation.cc:407] Connection header received
0810 15:24:39.336788 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.336790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.336804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.336857 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:39.337548 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.338102 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.339172 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.339382 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.339465 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.342020 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:39.342022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:39.342022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:39.342031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:39.343355 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.343797 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.343798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.343798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.343863 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.344126 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.344127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.344127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.344253 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:39.344349 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.344727 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.344822 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:24:39.345232 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.335333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:32883 (local address 127.4.64.67:45233)
0810 15:24:39.335556 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.335604 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.336650 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.336936 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.336940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.337226 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:39.337432 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.337436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.338209 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.338210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.339097 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.339100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.339524 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.339528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.339676 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.340321 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:39.340340 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:39.341919 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:39.343449 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:39.343457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:39.343459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:39.343705 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:39.343934 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:39.343935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:39.343935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:39.344054 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:39.344332 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:39.344341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:39.344543 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.344698 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.344896 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260810 15:24:39.345927 25821 heartbeater.cc:347] Connected to a master server at 127.4.64.126:32883
I20260810 15:24:39.346021 25821 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:39.346220 25821 heartbeater.cc:511] Master 127.4.64.126:32883 requested a full tablet report, sending...
I20260810 15:24:39.346710 25309 ts_manager.cc:194] Registered new tserver with Master: d9be2965b7cf4226a862822d1c417fbe (127.4.64.67:37883)
I20260810 15:24:39.347223 25309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:45233
I20260810 15:24:39.351934 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:24:39.361191 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.353660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.4.64.126:32883)
0810 15:24:39.353721 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.353723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.353972 (+ 249us) server_negotiation.cc:407] Connection header received
0810 15:24:39.354074 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.354075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.354088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.354174 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:39.354912 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.355454 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.356287 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.356506 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.356594 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.358413 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:39.358415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:39.358415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:39.358424 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:39.359854 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.360310 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.360311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.360311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.360373 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:39.360606 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:39.360607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:39.360608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:39.360761 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:39.360881 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.361012 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.361079 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:24:39.363178 25309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:24:39.365046 25309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:24:39.372617 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.367925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.4.64.67:37883)
0810 15:24:39.368349 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.368355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.368365 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:39.368410 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.368413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.368430 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.368510 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:39.369006 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.370041 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.371593 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.372173 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.372259 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.372323 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.372482 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.372525 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":92,"threads_started":1}
I20260810 15:24:39.372604 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.367799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37883 (local address 127.0.0.1:49258)
0810 15:24:39.368078 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.368122 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.368206 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.368545 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.368547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.368549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:39.368625 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.368626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.370151 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.370152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.371473 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.371476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.372257 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.372258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.372350 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.372364 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.372464 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.372526 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":92,"threads_started":1}
I20260810 15:24:39.372879 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.367801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:32927 (local address 127.0.0.1:39524)
0810 15:24:39.368319 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.368329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.368366 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.368605 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.368606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.368609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:39.368661 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.368664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.370382 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.370383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.371527 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.371529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.372430 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.372431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.372500 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.372509 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.372556 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.372604 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":93,"thread_start_us":65,"threads_started":1}
I20260810 15:24:39.373044 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.367802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:36713 (local address 127.0.0.1:60110)
0810 15:24:39.368436 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.368444 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.368481 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.368928 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.368929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.368931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:39.368969 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.368971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.370600 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.370601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.371421 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.371424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.372428 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.372430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.372500 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.372504 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.372556 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.372604 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":6,"thread_start_us":122,"threads_started":1}
I20260810 15:24:39.372816 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.368135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.4.64.66:36713)
0810 15:24:39.368749 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.368757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.368766 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:24:39.368822 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.368824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.368837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.368910 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:39.369066 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.370483 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.371518 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.372259 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.372334 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.372389 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.372664 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.372704 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":102,"threads_started":1}
I20260810 15:24:39.373381 25748 tablet_service.cc:1467] Processing CreateTablet for tablet b90c5190423e43d2902396f4fa354043 (DEFAULT_TABLE table=test-table [id=64caec3ed61f4f8e97be3bb71cbffc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:39.373677 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.368085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.4.64.65:32927)
0810 15:24:39.368349 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.368354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.368369 (+ 15us) server_negotiation.cc:407] Connection header received
0810 15:24:39.368436 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.368438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.368455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.368543 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:39.368818 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.370265 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.371592 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.372271 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.372334 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.372389 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.372651 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.373543 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
I20260810 15:24:39.373818 25748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90c5190423e43d2902396f4fa354043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:39.373860 25597 tablet_service.cc:1467] Processing CreateTablet for tablet b90c5190423e43d2902396f4fa354043 (DEFAULT_TABLE table=test-table [id=64caec3ed61f4f8e97be3bb71cbffc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:39.374198 25597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90c5190423e43d2902396f4fa354043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:39.374276 25452 tablet_service.cc:1467] Processing CreateTablet for tablet b90c5190423e43d2902396f4fa354043 (DEFAULT_TABLE table=test-table [id=64caec3ed61f4f8e97be3bb71cbffc9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:39.374604 25452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90c5190423e43d2902396f4fa354043. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:39.377130 25839 tablet_bootstrap.cc:492] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: Bootstrap starting.
I20260810 15:24:39.377516 25841 tablet_bootstrap.cc:492] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Bootstrap starting.
I20260810 15:24:39.377804 25840 tablet_bootstrap.cc:492] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Bootstrap starting.
I20260810 15:24:39.377923 25839 tablet_bootstrap.cc:654] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:39.378280 25841 tablet_bootstrap.cc:654] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:39.378471 25839 log.cc:824] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:39.378530 25840 tablet_bootstrap.cc:654] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:39.378957 25841 log.cc:824] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:39.379149 25840 log.cc:824] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:39.379276 25839 tablet_bootstrap.cc:492] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: No bootstrap required, opened a new log
I20260810 15:24:39.379356 25839 ts_tablet_manager.cc:1397] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:24:39.379743 25841 tablet_bootstrap.cc:492] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: No bootstrap required, opened a new log
I20260810 15:24:39.379819 25841 ts_tablet_manager.cc:1397] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:24:39.380026 25840 tablet_bootstrap.cc:492] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: No bootstrap required, opened a new log
I20260810 15:24:39.380095 25840 ts_tablet_manager.cc:1397] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:24:39.380995 25839 raft_consensus.cc:348] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.381168 25839 raft_consensus.cc:374] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:39.381212 25839 raft_consensus.cc:729] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc94074ceb754433919a91d036233084, State: Initialized, Role: FOLLOWER
I20260810 15:24:39.381327 25839 consensus_queue.cc:260] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.381667 25839 ts_tablet_manager.cc:1428] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:24:39.381667 25523 heartbeater.cc:503] Master 127.4.64.126:32883 was elected leader, sending a full tablet report...
I20260810 15:24:39.382051 25841 raft_consensus.cc:348] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.382229 25841 raft_consensus.cc:374] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:39.382266 25841 raft_consensus.cc:729] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0f5e86958c4dcb9066c680bd0fbc52, State: Initialized, Role: FOLLOWER
I20260810 15:24:39.382488 25841 consensus_queue.cc:260] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.382597 25840 raft_consensus.cc:348] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.382774 25840 raft_consensus.cc:374] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:39.382824 25840 raft_consensus.cc:729] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9be2965b7cf4226a862822d1c417fbe, State: Initialized, Role: FOLLOWER
I20260810 15:24:39.382881 25841 ts_tablet_manager.cc:1428] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 15:24:39.382946 25840 consensus_queue.cc:260] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.382979 25673 heartbeater.cc:503] Master 127.4.64.126:32883 was elected leader, sending a full tablet report...
I20260810 15:24:39.383226 25840 ts_tablet_manager.cc:1428] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:24:39.383235 25821 heartbeater.cc:503] Master 127.4.64.126:32883 was elected leader, sending a full tablet report...
W20260810 15:24:39.413771 25674 tablet.cc:2369] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:24:39.496114 25524 tablet.cc:2369] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:39.525600 25846 raft_consensus.cc:482] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:39.525808 25846 raft_consensus.cc:504] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.526335 25846 leader_election.cc:288] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9be2965b7cf4226a862822d1c417fbe (127.4.64.67:37883), bc94074ceb754433919a91d036233084 (127.4.64.65:32927)
I20260810 15:24:39.529346 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.526666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:59563 (local address 127.4.64.65:32927)
0810 15:24:39.526721 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.526723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.526858 (+ 135us) server_negotiation.cc:407] Connection header received
0810 15:24:39.526894 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.526895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.526908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.526983 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:39.527152 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.527620 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.528584 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.528958 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.529046 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.529058 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.529237 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.529274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:24:39.529361 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.526556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:32927 (local address 127.4.64.66:59563)
0810 15:24:39.526783 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.526792 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.526839 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.526987 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.526988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.526989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:39.527070 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.527072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.527718 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.527719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.528486 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.528488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.529058 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.529060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.529137 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.529147 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.529227 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.529272 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":44,"thread_start_us":135,"threads_started":1}
I20260810 15:24:39.529413 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.526632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:41505 (local address 127.4.64.67:37883)
0810 15:24:39.526681 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.526683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.526695 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:24:39.526729 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.526730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.526742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.526837 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:39.527011 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.527552 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.528547 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.529038 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.529120 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.529131 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.529299 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.529338 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:24:39.529449 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.526535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37883 (local address 127.4.64.66:41505)
0810 15:24:39.526594 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.526608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.526662 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.526843 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.526844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.526845 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:39.526927 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.526929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.527648 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.527649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.528442 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.528445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.529132 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.529133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.529220 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.529223 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.529291 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.529335 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:24:39.529811 25474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90c5190423e43d2902396f4fa354043" candidate_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc94074ceb754433919a91d036233084" is_pre_election: true
I20260810 15:24:39.529814 25768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90c5190423e43d2902396f4fa354043" candidate_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9be2965b7cf4226a862822d1c417fbe" is_pre_election: true
I20260810 15:24:39.529991 25474 raft_consensus.cc:2454] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0f5e86958c4dcb9066c680bd0fbc52 in term 0.
I20260810 15:24:39.530005 25768 raft_consensus.cc:2454] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0f5e86958c4dcb9066c680bd0fbc52 in term 0.
I20260810 15:24:39.530242 25547 leader_election.cc:302] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0f5e86958c4dcb9066c680bd0fbc52, d9be2965b7cf4226a862822d1c417fbe; no voters:
I20260810 15:24:39.530378 25846 raft_consensus.cc:2781] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:24:39.530435 25846 raft_consensus.cc:482] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:24:39.530467 25846 raft_consensus.cc:3037] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:39.531201 25846 raft_consensus.cc:504] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.531359 25846 leader_election.cc:288] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [CANDIDATE]: Term 1 election: Requested vote from peers d9be2965b7cf4226a862822d1c417fbe (127.4.64.67:37883), bc94074ceb754433919a91d036233084 (127.4.64.65:32927)
I20260810 15:24:39.531543 25474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90c5190423e43d2902396f4fa354043" candidate_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc94074ceb754433919a91d036233084"
I20260810 15:24:39.531601 25768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90c5190423e43d2902396f4fa354043" candidate_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9be2965b7cf4226a862822d1c417fbe"
I20260810 15:24:39.531622 25474 raft_consensus.cc:3037] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:39.531683 25768 raft_consensus.cc:3037] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:39.532409 25768 raft_consensus.cc:2454] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0f5e86958c4dcb9066c680bd0fbc52 in term 1.
I20260810 15:24:39.532428 25474 raft_consensus.cc:2454] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0f5e86958c4dcb9066c680bd0fbc52 in term 1.
I20260810 15:24:39.532585 25547 leader_election.cc:302] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c0f5e86958c4dcb9066c680bd0fbc52, d9be2965b7cf4226a862822d1c417fbe; no voters:
I20260810 15:24:39.532711 25846 raft_consensus.cc:2781] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:39.532894 25846 raft_consensus.cc:686] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0f5e86958c4dcb9066c680bd0fbc52, State: Running, Role: LEADER
I20260810 15:24:39.533003 25846 consensus_queue.cc:237] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } }
I20260810 15:24:39.533970 25309 catalog_manager.cc:5495] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0f5e86958c4dcb9066c680bd0fbc52 (127.4.64.66). New cstate: current_term: 1 leader_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c0f5e86958c4dcb9066c680bd0fbc52" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 } health_report { overall_health: UNKNOWN } } }
I20260810 15:24:39.556768 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.554262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.4.64.66:36713)
0810 15:24:39.554342 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.554344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.554351 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:24:39.554389 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.554390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.554412 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.554497 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:24:39.554659 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.555142 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.555957 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.556156 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.556233 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.556397 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:24:39.556501 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:24:39.556621 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.556658 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.556669 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 15:24:39.558674 25768 raft_consensus.cc:1261] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 1 FOLLOWER]: Refusing update from remote peer 7c0f5e86958c4dcb9066c680bd0fbc52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:39.558712 25474 raft_consensus.cc:1261] T b90c5190423e43d2902396f4fa354043 P bc94074ceb754433919a91d036233084 [term 1 FOLLOWER]: Refusing update from remote peer 7c0f5e86958c4dcb9066c680bd0fbc52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:39.559073 25850 consensus_queue.cc:1035] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9be2965b7cf4226a862822d1c417fbe" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:39.559249 25846 consensus_queue.cc:1035] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc94074ceb754433919a91d036233084" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:39.562382 25855 mvcc.cc:205] Tried to move back new op lower bound from 7316993964269543424 to 7316993964168056832. Current Snapshot: MvccSnapshot[applied={T|T < 7316993964269543424}]
I20260810 15:24:39.564247 25309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:39.564369 25309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:39.565366 25309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49fb3b3e88b74c2e9d3fe6186c28daac: Sending DeleteTablet for 3 replicas of tablet b90c5190423e43d2902396f4fa354043
I20260810 15:24:39.565788 25748 tablet_service.cc:1514] Processing DeleteTablet for tablet b90c5190423e43d2902396f4fa354043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:49258
I20260810 15:24:39.565901 25452 tablet_service.cc:1514] Processing DeleteTablet for tablet b90c5190423e43d2902396f4fa354043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:39524
I20260810 15:24:39.566260 25864 tablet_replica.cc:330] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: stopping tablet replica
I20260810 15:24:39.566241 25597 tablet_service.cc:1514] Processing DeleteTablet for tablet b90c5190423e43d2902396f4fa354043 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:39 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:60110
I20260810 15:24:39.566406 25864 raft_consensus.cc:2229] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:39.566495 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25379
I20260810 15:24:39.566787 25866 tablet_replica.cc:330] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: stopping tablet replica
I20260810 15:24:39.566896 25866 raft_consensus.cc:2229] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:24:39.566994 25866 raft_consensus.cc:2258] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:39.568463 25866 ts_tablet_manager.cc:1905] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:39.569221 25864 raft_consensus.cc:2258] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:39.570137 25864 ts_tablet_manager.cc:1905] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:39.570696 25866 ts_tablet_manager.cc:1918] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:39.570832 25866 log.cc:1196] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/b90c5190423e43d2902396f4fa354043
I20260810 15:24:39.571219 25866 ts_tablet_manager.cc:1939] T b90c5190423e43d2902396f4fa354043 P 7c0f5e86958c4dcb9066c680bd0fbc52: Deleting consensus metadata
I20260810 15:24:39.571724 25292 catalog_manager.cc:4841] TS 7c0f5e86958c4dcb9066c680bd0fbc52 (127.4.64.66:36713): tablet b90c5190423e43d2902396f4fa354043 (table test-table [id=64caec3ed61f4f8e97be3bb71cbffc9f]) successfully deleted
I20260810 15:24:39.572008 25864 ts_tablet_manager.cc:1918] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:39.572105 25864 log.cc:1196] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/b90c5190423e43d2902396f4fa354043
I20260810 15:24:39.572386 25864 ts_tablet_manager.cc:1939] T b90c5190423e43d2902396f4fa354043 P d9be2965b7cf4226a862822d1c417fbe: Deleting consensus metadata
I20260810 15:24:39.572872 25294 catalog_manager.cc:4841] TS d9be2965b7cf4226a862822d1c417fbe (127.4.64.67:37883): tablet b90c5190423e43d2902396f4fa354043 (table test-table [id=64caec3ed61f4f8e97be3bb71cbffc9f]) successfully deleted
W20260810 15:24:39.582779 25291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:32927 (error 108)
I20260810 15:24:39.583050 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25528
I20260810 15:24:39.585482 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.584959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:32927 (local address 127.0.0.1:39536)
0810 15:24:39.585380 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.585433 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394}
W20260810 15:24:39.585668 25291 catalog_manager.cc:4584] TS bc94074ceb754433919a91d036233084 (127.4.64.65:32927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b90c5190423e43d2902396f4fa354043: Network error: Client connection negotiation failed: client connection to 127.4.64.65:32927: connect: Connection refused (error 111)
I20260810 15:24:39.592743 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25678
I20260810 15:24:39.600311 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25273
2026-08-10T15:24:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T15:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:39Z Disabled control of system clock
I20260810 15:24:39.634310 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:38185
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:39.719666 25871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:39.719892 25871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:39.719913 25871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:39.721004 25871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:39.721038 25871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:39.721051 25871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:39.721063 25871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:39.721076 25871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:39.722745 25871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:38185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:39.723039 25871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:39.723389 25871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:39.728742 25876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.728807 25879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.728807 25877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:39.729035 25871 server_base.cc:1034] running on GCE node
I20260810 15:24:39.729362 25871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:39.729773 25871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:39.730974 25871 hybrid_clock.cc:648] HybridClock initialized: now 1786375479730926 us; error 60 us; skew 500 ppm
I20260810 15:24:39.736347 25871 webserver.cc:460] Webserver started at http://127.4.64.126:34939/ using document root <none> and password file <none>
I20260810 15:24:39.736706 25871 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:39.736794 25871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:39.736974 25871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:39.738327 25871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "f43b951b07fe4553aaa33d2dff96135f"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "aada465dc82b08c532516c604984448f"
server_key_iv: "656996634f2eace93ed4a9d711b16ff5"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.738747 25871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "f43b951b07fe4553aaa33d2dff96135f"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "aada465dc82b08c532516c604984448f"
server_key_iv: "656996634f2eace93ed4a9d711b16ff5"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.740242 25871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:24:39.741261 25889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.741598 25871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:39.741796 25871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "f43b951b07fe4553aaa33d2dff96135f"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "aada465dc82b08c532516c604984448f"
server_key_iv: "656996634f2eace93ed4a9d711b16ff5"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.741869 25871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:39.790498 25871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:39.790838 25871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:39.790956 25871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:39.796134 25871 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:38185
I20260810 15:24:39.796161 25951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:38185 every 8 connection(s)
I20260810 15:24:39.796586 25871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:39.798079 25952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:39.800153 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25871
I20260810 15:24:39.800292 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:39.800525 4353 external_mini_cluster.cc:1428] Setting key 80f06c77e20122ef187b464a63ae6ea5
I20260810 15:24:39.800992 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: Bootstrap starting.
I20260810 15:24:39.801748 25952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:39.802073 25952 log.cc:824] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:39.802901 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: No bootstrap required, opened a new log
I20260810 15:24:39.805192 25952 raft_consensus.cc:348] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43b951b07fe4553aaa33d2dff96135f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38185 } }
I20260810 15:24:39.805399 25952 raft_consensus.cc:374] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:39.805444 25952 raft_consensus.cc:729] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43b951b07fe4553aaa33d2dff96135f, State: Initialized, Role: FOLLOWER
I20260810 15:24:39.805665 25952 consensus_queue.cc:260] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43b951b07fe4553aaa33d2dff96135f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38185 } }
I20260810 15:24:39.805783 25952 raft_consensus.cc:388] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:39.805830 25952 raft_consensus.cc:482] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:39.805950 25952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:39.806675 25952 raft_consensus.cc:504] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43b951b07fe4553aaa33d2dff96135f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38185 } }
I20260810 15:24:39.806818 25952 leader_election.cc:302] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f43b951b07fe4553aaa33d2dff96135f; no voters:
I20260810 15:24:39.807009 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.801124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.4.64.126:38185)
0810 15:24:39.801362 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.801370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.801397 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:24:39.801915 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.801925 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.802006 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.802130 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:39.802423 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.803470 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.804668 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.805209 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.805347 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.806166 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:39.806170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:39.806170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:39.806362 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:39.806399 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:39.806411 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:39.806473 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.806582 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.806665 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260810 15:24:39.807062 25952 leader_election.cc:288] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:39.807094 25957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:39.807212 25957 raft_consensus.cc:686] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [term 1 LEADER]: Becoming Leader. State: Replica: f43b951b07fe4553aaa33d2dff96135f, State: Running, Role: LEADER
I20260810 15:24:39.807335 25952 sys_catalog.cc:564] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:39.807315 25957 consensus_queue.cc:237] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43b951b07fe4553aaa33d2dff96135f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38185 } }
I20260810 15:24:39.808069 25958 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f43b951b07fe4553aaa33d2dff96135f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43b951b07fe4553aaa33d2dff96135f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38185 } } }
I20260810 15:24:39.808071 25959 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f43b951b07fe4553aaa33d2dff96135f. Latest consensus state: current_term: 1 leader_uuid: "f43b951b07fe4553aaa33d2dff96135f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43b951b07fe4553aaa33d2dff96135f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 38185 } } }
I20260810 15:24:39.808245 25958 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:39.808257 25959 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f [sys.catalog]: This master's current role is: LEADER
W20260810 15:24:39.809476 25974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:24:39.809526 25974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:24:39.809602 25967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:39.809881 25967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:39.811514 25967 catalog_manager.cc:1347] Generated new cluster ID: 96bb91f87b0b4ac49b71cc603b7f0d3d
I20260810 15:24:39.811575 25967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:39.829054 25967 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:39.829777 25967 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:39.833328 25967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: Generated new TSK 0
I20260810 15:24:39.833549 25967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:39.842938 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38185
--builtin_ntp_servers=127.4.64.84:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:39.933727 25977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:39.933943 25977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:39.933965 25977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:39.935155 25977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:39.935204 25977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:39.935228 25977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:39.936651 25977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:39.936918 25977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:39.937242 25977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:39.943783 25983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.943876 25982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:39.943780 25985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:39.944128 25977 server_base.cc:1034] running on GCE node
I20260810 15:24:39.944389 25977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:39.944658 25977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:39.945827 25977 hybrid_clock.cc:648] HybridClock initialized: now 1786375479945791 us; error 55 us; skew 500 ppm
I20260810 15:24:39.947245 25977 webserver.cc:460] Webserver started at http://127.4.64.65:38669/ using document root <none> and password file <none>
I20260810 15:24:39.947525 25977 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:39.947600 25977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:39.947747 25977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:39.948853 25977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "7e5f414f3e7248d184e6388ca94eedec"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "4bc77d57bd6f11fd3623e23c5a052264"
server_key_iv: "988172fccd8e9df8ba11be6cb961495e"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.949259 25977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "7e5f414f3e7248d184e6388ca94eedec"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "4bc77d57bd6f11fd3623e23c5a052264"
server_key_iv: "988172fccd8e9df8ba11be6cb961495e"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.950778 25977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:39.951629 25992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.951870 25977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:39.951974 25977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "7e5f414f3e7248d184e6388ca94eedec"
format_stamp: "Formatted at 2026-08-10 15:24:39 on dist-test-slave-9zdj"
server_key: "4bc77d57bd6f11fd3623e23c5a052264"
server_key_iv: "988172fccd8e9df8ba11be6cb961495e"
server_key_version: "encryptionkey@0"
I20260810 15:24:39.952059 25977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:39.964550 25977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:39.964902 25977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:39.965066 25977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:39.965349 25977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:39.965785 25977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:39.965847 25977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.965890 25977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:39.965919 25977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:39.972560 25977 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:42481
I20260810 15:24:39.972616 26116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:42481 every 8 connection(s)
I20260810 15:24:39.972970 25977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:24:39.978564 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.973730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:34451 (local address 127.4.64.126:38185)
0810 15:24:39.973789 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:39.973791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:39.974100 (+ 309us) server_negotiation.cc:407] Connection header received
0810 15:24:39.974639 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:39.974641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:39.974654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:39.974709 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:39.975520 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.976037 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.977072 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.977277 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.977360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.978241 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:39.978243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:39.978243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:39.978248 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:39.978265 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:39.978270 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:39.978352 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:39.978451 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 15:24:39.978480 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:24:39.978790 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:39.973775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:38185 (local address 127.4.64.65:34451)
0810 15:24:39.973962 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:39.973998 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:39.974526 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:39.974777 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:39.974781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:39.974791 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:39.975425 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.975431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.976147 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.976148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.976972 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:39.976974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:39.977369 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:39.977371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:39.977521 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:39.978093 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:39.978093 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:39.978124 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:39.978146 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:39.978356 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:39.978363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:39.978366 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:39.978453 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:24:39.978561 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260810 15:24:39.979360 26117 heartbeater.cc:347] Connected to a master server at 127.4.64.126:38185
I20260810 15:24:39.979462 26117 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:39.979611 26117 heartbeater.cc:511] Master 127.4.64.126:38185 requested a full tablet report, sending...
I20260810 15:24:39.979708 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 25977
I20260810 15:24:39.979802 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:24:39.980080 4353 external_mini_cluster.cc:1428] Setting key 61ed577d97453bd71c09c816702f084e
I20260810 15:24:39.980244 25907 ts_manager.cc:194] Registered new tserver with Master: 7e5f414f3e7248d184e6388ca94eedec (127.4.64.65:42481)
I20260810 15:24:39.981138 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.65:34451
I20260810 15:24:39.981210 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38185
--builtin_ntp_servers=127.4.64.84:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:40.074679 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:40.074908 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:40.074934 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:40.076452 26121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:40.076509 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:40.076538 26121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:40.078174 26121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:40.078457 26121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:40.078842 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:40.084868 26130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:40.085031 26121 server_base.cc:1034] running on GCE node
W20260810 15:24:40.084877 26127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:40.084875 26126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:40.085420 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:40.085759 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:40.086956 26121 hybrid_clock.cc:648] HybridClock initialized: now 1786375480086951 us; error 41 us; skew 500 ppm
I20260810 15:24:40.088519 26121 webserver.cc:460] Webserver started at http://127.4.64.66:36537/ using document root <none> and password file <none>
I20260810 15:24:40.088902 26121 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:40.088995 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:40.089160 26121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:40.090379 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "8febdecccbb54a4ebde27e121436ef09"
format_stamp: "Formatted at 2026-08-10 15:24:40 on dist-test-slave-9zdj"
server_key: "8122de3b982df142c99668c7cb0d4913"
server_key_iv: "7c8388c8412340b310eea89608688218"
server_key_version: "encryptionkey@0"
I20260810 15:24:40.090770 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "8febdecccbb54a4ebde27e121436ef09"
format_stamp: "Formatted at 2026-08-10 15:24:40 on dist-test-slave-9zdj"
server_key: "8122de3b982df142c99668c7cb0d4913"
server_key_iv: "7c8388c8412340b310eea89608688218"
server_key_version: "encryptionkey@0"
I20260810 15:24:40.092173 26121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:24:40.092967 26137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:40.093151 26121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:40.093261 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "8febdecccbb54a4ebde27e121436ef09"
format_stamp: "Formatted at 2026-08-10 15:24:40 on dist-test-slave-9zdj"
server_key: "8122de3b982df142c99668c7cb0d4913"
server_key_iv: "7c8388c8412340b310eea89608688218"
server_key_version: "encryptionkey@0"
I20260810 15:24:40.093344 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:40.101264 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:40.101598 26121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:40.101816 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:40.102066 26121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:40.102435 26121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:40.102494 26121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:40.102530 26121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:40.102556 26121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:40.109131 26121 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:34141
I20260810 15:24:40.109189 26264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:34141 every 8 connection(s)
I20260810 15:24:40.109551 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:40.115624 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.110419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:57247 (local address 127.4.64.126:38185)
0810 15:24:40.110502 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.110504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.110751 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:24:40.111266 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.111267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.111278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.111343 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:40.112088 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.112677 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.113813 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.114014 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.114093 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.115227 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:40.115229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:40.115229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:40.115234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:40.115249 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:40.115253 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:40.115307 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.115513 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.115551 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 15:24:40.115752 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.110393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:38185 (local address 127.4.64.66:57247)
0810 15:24:40.110618 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.110656 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.111151 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.111445 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.111449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.111458 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:40.111994 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.111999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.112797 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.112798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.113722 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.113725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.114133 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.114134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.114341 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.114969 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:40.114970 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:40.115005 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:40.115031 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:40.115374 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:40.115381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:40.115384 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.115467 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.115565 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260810 15:24:40.116382 26265 heartbeater.cc:347] Connected to a master server at 127.4.64.126:38185
I20260810 15:24:40.116492 26265 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:40.116757 26265 heartbeater.cc:511] Master 127.4.64.126:38185 requested a full tablet report, sending...
I20260810 15:24:40.117355 25907 ts_manager.cc:194] Registered new tserver with Master: 8febdecccbb54a4ebde27e121436ef09 (127.4.64.66:34141)
I20260810 15:24:40.117712 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 26121
I20260810 15:24:40.117835 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:24:40.117995 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.66:57247
I20260810 15:24:40.118113 4353 external_mini_cluster.cc:1428] Setting key ab08f411b207db68e3bc42ede1276339
I20260810 15:24:40.119385 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38185
--builtin_ntp_servers=127.4.64.84:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:40.214412 26269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:40.214672 26269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:40.214699 26269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:40.216053 26269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:40.216105 26269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:40.216135 26269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:40.217731 26269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:38185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:40.218036 26269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:40.218405 26269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:40.224720 26274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:40.224781 26275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:40.224833 26277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:40.225106 26269 server_base.cc:1034] running on GCE node
I20260810 15:24:40.225386 26269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:40.225718 26269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:40.226903 26269 hybrid_clock.cc:648] HybridClock initialized: now 1786375480226891 us; error 30 us; skew 500 ppm
I20260810 15:24:40.228343 26269 webserver.cc:460] Webserver started at http://127.4.64.67:33377/ using document root <none> and password file <none>
I20260810 15:24:40.228629 26269 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:40.228681 26269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:40.228870 26269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:40.230062 26269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "3b5c362788cc4a83b4b76426766a5e12"
format_stamp: "Formatted at 2026-08-10 15:24:40 on dist-test-slave-9zdj"
server_key: "5d01e6d11b70d4847d6323f2d5c7934a"
server_key_iv: "f66fbc84eb161bf3feae83ec8a0f5492"
server_key_version: "encryptionkey@0"
I20260810 15:24:40.230432 26269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "3b5c362788cc4a83b4b76426766a5e12"
format_stamp: "Formatted at 2026-08-10 15:24:40 on dist-test-slave-9zdj"
server_key: "5d01e6d11b70d4847d6323f2d5c7934a"
server_key_iv: "f66fbc84eb161bf3feae83ec8a0f5492"
server_key_version: "encryptionkey@0"
I20260810 15:24:40.231884 26269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:40.232721 26283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:40.232959 26269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:40.233057 26269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "3b5c362788cc4a83b4b76426766a5e12"
format_stamp: "Formatted at 2026-08-10 15:24:40 on dist-test-slave-9zdj"
server_key: "5d01e6d11b70d4847d6323f2d5c7934a"
server_key_iv: "f66fbc84eb161bf3feae83ec8a0f5492"
server_key_version: "encryptionkey@0"
I20260810 15:24:40.233131 26269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:40.261032 26269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:40.261396 26269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:40.261555 26269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:40.261905 26269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:40.262310 26269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:40.262377 26269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:40.262413 26269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:40.262440 26269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:40.268831 26269 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:37927
I20260810 15:24:40.268882 26417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:37927 every 8 connection(s)
I20260810 15:24:40.269279 26269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:40.275094 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.270336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:59069 (local address 127.4.64.126:38185)
0810 15:24:40.270417 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.270420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.270566 (+ 146us) server_negotiation.cc:407] Connection header received
0810 15:24:40.271071 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.271072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.271084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.271153 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:40.271892 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.272453 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.273544 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.273817 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.273889 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.274845 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:40.274846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:40.274846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:40.274851 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:40.274865 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:40.274869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:40.274921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.275000 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.275024 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 15:24:40.275367 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.270196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:38185 (local address 127.4.64.67:59069)
0810 15:24:40.270411 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.270452 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.270979 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.271226 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.271230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.271238 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:40.271796 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.271800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.272560 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.272561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.273434 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.273437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.273940 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.273941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.274079 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.274645 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:40.274646 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:40.274676 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:40.274703 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:40.274929 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:40.274932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:40.274934 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.274986 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.275078 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260810 15:24:40.276068 26419 heartbeater.cc:347] Connected to a master server at 127.4.64.126:38185
I20260810 15:24:40.276266 26419 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:40.276537 26419 heartbeater.cc:511] Master 127.4.64.126:38185 requested a full tablet report, sending...
I20260810 15:24:40.276528 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 26269
I20260810 15:24:40.276620 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:24:40.276887 4353 external_mini_cluster.cc:1428] Setting key 772bccfb315afeae574909d8ffedb960
I20260810 15:24:40.277016 25907 ts_manager.cc:194] Registered new tserver with Master: 3b5c362788cc4a83b4b76426766a5e12 (127.4.64.67:37927)
I20260810 15:24:40.277524 25907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.67:59069
I20260810 15:24:40.277993 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:24:40.282930 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.279608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.4.64.126:38185)
0810 15:24:40.279673 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.279675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.279817 (+ 142us) server_negotiation.cc:407] Connection header received
0810 15:24:40.279840 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.279841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.279850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.279906 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:40.280050 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.280517 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.281325 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.281526 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.281596 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.282618 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:40.282620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:40.282620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:40.282624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:40.282639 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:40.282642 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:40.282751 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.282798 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.282808 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:24:40.284895 25906 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:24:40.286600 25906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:24:40.293447 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.289257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.4.64.66:34141)
0810 15:24:40.289448 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.289453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.289469 (+ 16us) server_negotiation.cc:407] Connection header received
0810 15:24:40.289542 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.289544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.289560 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.289826 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:40.290095 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.290926 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.292084 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.292876 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.292947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.293044 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.293304 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.293353 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":121,"threads_started":1}
I20260810 15:24:40.293550 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.289390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:42481 (local address 127.0.0.1:43292)
0810 15:24:40.289797 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.289809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.289858 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.290280 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.290281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.290284 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:40.290323 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.290324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.291587 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.291589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.292539 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.292541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.293280 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.293281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.293348 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.293352 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.293383 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.293424 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":58,"threads_started":1}
I20260810 15:24:40.293435 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.289101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:34141 (local address 127.0.0.1:55500)
0810 15:24:40.289343 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.289386 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.289451 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.289912 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.289915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.289918 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:40.290005 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.290008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.291021 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.291022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.291989 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.291992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.293081 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.293082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.293169 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.293184 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.293286 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.293353 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
I20260810 15:24:40.293778 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.289577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.4.64.65:42481)
0810 15:24:40.290123 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.290128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.290139 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:24:40.290181 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.290183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.290199 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.290283 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:40.290386 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.291484 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.292634 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.293169 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.293489 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.293590 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.293653 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.293690 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
I20260810 15:24:40.294723 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.289569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37927 (local address 127.0.0.1:53424)
0810 15:24:40.289890 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.289900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.289947 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.290162 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.290163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.290165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:40.290224 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.290226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.291558 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.291559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.293074 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.293077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.294492 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.294493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.294555 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.294559 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.294609 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.294653 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":422,"thread_start_us":106,"threads_started":1}
I20260810 15:24:40.294646 26046 tablet_service.cc:1467] Processing CreateTablet for tablet ce0c969f476b42158df083105e8d1ee0 (DEFAULT_TABLE table=test-table [id=2dd1fa3d1a044ddea9d803999868c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:40.294654 26192 tablet_service.cc:1467] Processing CreateTablet for tablet ce0c969f476b42158df083105e8d1ee0 (DEFAULT_TABLE table=test-table [id=2dd1fa3d1a044ddea9d803999868c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:40.295055 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.289706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.4.64.67:37927)
0810 15:24:40.289999 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.290004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.290014 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:40.290062 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.290064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.290081 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.290175 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:40.290312 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.291458 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.293185 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.294303 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.294376 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.294454 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.294903 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.294946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":87,"threads_started":1}
I20260810 15:24:40.295081 26046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0c969f476b42158df083105e8d1ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:40.295079 26192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0c969f476b42158df083105e8d1ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:40.295673 26343 tablet_service.cc:1467] Processing CreateTablet for tablet ce0c969f476b42158df083105e8d1ee0 (DEFAULT_TABLE table=test-table [id=2dd1fa3d1a044ddea9d803999868c0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:40.296025 26343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0c969f476b42158df083105e8d1ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:40.298326 26436 tablet_bootstrap.cc:492] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Bootstrap starting.
I20260810 15:24:40.298687 26437 tablet_bootstrap.cc:492] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: Bootstrap starting.
I20260810 15:24:40.299363 26436 tablet_bootstrap.cc:654] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:40.299376 26437 tablet_bootstrap.cc:654] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:40.299443 26438 tablet_bootstrap.cc:492] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Bootstrap starting.
I20260810 15:24:40.299865 26437 log.cc:824] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:40.299913 26436 log.cc:824] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:40.300156 26438 tablet_bootstrap.cc:654] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:40.300488 26438 log.cc:824] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:40.300799 26437 tablet_bootstrap.cc:492] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: No bootstrap required, opened a new log
I20260810 15:24:40.300844 26436 tablet_bootstrap.cc:492] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: No bootstrap required, opened a new log
I20260810 15:24:40.300897 26437 ts_tablet_manager.cc:1397] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:24:40.300913 26436 ts_tablet_manager.cc:1397] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:24:40.301251 26438 tablet_bootstrap.cc:492] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: No bootstrap required, opened a new log
I20260810 15:24:40.301337 26438 ts_tablet_manager.cc:1397] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:24:40.303130 26437 raft_consensus.cc:348] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.303303 26437 raft_consensus.cc:374] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:40.303365 26437 raft_consensus.cc:729] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5f414f3e7248d184e6388ca94eedec, State: Initialized, Role: FOLLOWER
I20260810 15:24:40.303494 26437 consensus_queue.cc:260] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.303769 26437 ts_tablet_manager.cc:1428] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:24:40.303654 26436 raft_consensus.cc:348] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.303826 26436 raft_consensus.cc:374] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:40.303905 26436 raft_consensus.cc:729] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8febdecccbb54a4ebde27e121436ef09, State: Initialized, Role: FOLLOWER
I20260810 15:24:40.303920 26117 heartbeater.cc:503] Master 127.4.64.126:38185 was elected leader, sending a full tablet report...
I20260810 15:24:40.304064 26436 consensus_queue.cc:260] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.304044 26438 raft_consensus.cc:348] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.304209 26438 raft_consensus.cc:374] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:40.304265 26438 raft_consensus.cc:729] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5c362788cc4a83b4b76426766a5e12, State: Initialized, Role: FOLLOWER
I20260810 15:24:40.304395 26436 ts_tablet_manager.cc:1428] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:24:40.304395 26438 consensus_queue.cc:260] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.304524 26265 heartbeater.cc:503] Master 127.4.64.126:38185 was elected leader, sending a full tablet report...
I20260810 15:24:40.304656 26438 ts_tablet_manager.cc:1428] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:24:40.304780 26419 heartbeater.cc:503] Master 127.4.64.126:38185 was elected leader, sending a full tablet report...
W20260810 15:24:40.360322 26267 tablet.cc:2369] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:24:40.473816 26118 tablet.cc:2369] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:40.510594 26442 raft_consensus.cc:482] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:40.510772 26442 raft_consensus.cc:504] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.511270 26442 leader_election.cc:288] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8febdecccbb54a4ebde27e121436ef09 (127.4.64.66:34141), 3b5c362788cc4a83b4b76426766a5e12 (127.4.64.67:37927)
I20260810 15:24:40.515269 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.511873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:42991 (local address 127.4.64.67:37927)
0810 15:24:40.511920 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.511922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.511933 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:24:40.511997 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.511997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.512006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.512054 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:40.512267 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.512971 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.514384 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.514888 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.514934 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.514943 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.515145 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.515197 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:24:40.515326 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.511597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:36329 (local address 127.4.64.66:34141)
0810 15:24:40.511645 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.511648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.511696 (+ 48us) server_negotiation.cc:407] Connection header received
0810 15:24:40.511783 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.511784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.511798 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.511871 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:40.512102 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.512884 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.514407 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.514920 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.514966 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.514975 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.515210 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.515253 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:24:40.515513 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.511696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:37927 (local address 127.4.64.65:42991)
0810 15:24:40.511886 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.511896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.511939 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.512069 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.512070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.512071 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:40.512130 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.512132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.513074 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.513075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.514280 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.514283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.515002 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.515003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.515069 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.515074 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.515394 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.515437 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
I20260810 15:24:40.515882 26365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0c969f476b42158df083105e8d1ee0" candidate_uuid: "7e5f414f3e7248d184e6388ca94eedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5c362788cc4a83b4b76426766a5e12" is_pre_election: true
I20260810 15:24:40.515957 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.511473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:34141 (local address 127.4.64.65:36329)
0810 15:24:40.511606 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.511620 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:40.511718 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:40.511933 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:40.511934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:40.511936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:40.512015 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.512017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.513002 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.513003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.514297 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:40.514300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.515048 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.515049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:40.515121 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.515124 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:40.515828 (+ 704us) client_negotiation.cc:240] Negotiation successful
0810 15:24:40.515874 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":41,"thread_start_us":82,"threads_started":1}
I20260810 15:24:40.516042 26365 raft_consensus.cc:2454] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5f414f3e7248d184e6388ca94eedec in term 0.
I20260810 15:24:40.516273 25993 leader_election.cc:302] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5c362788cc4a83b4b76426766a5e12, 7e5f414f3e7248d184e6388ca94eedec; no voters:
I20260810 15:24:40.516470 26442 raft_consensus.cc:2781] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:24:40.516520 26442 raft_consensus.cc:482] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:24:40.516541 26442 raft_consensus.cc:3037] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:40.516614 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0c969f476b42158df083105e8d1ee0" candidate_uuid: "7e5f414f3e7248d184e6388ca94eedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8febdecccbb54a4ebde27e121436ef09" is_pre_election: true
I20260810 15:24:40.516747 26213 raft_consensus.cc:2454] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5f414f3e7248d184e6388ca94eedec in term 0.
I20260810 15:24:40.517467 26442 raft_consensus.cc:504] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
I20260810 15:24:40.517599 26442 leader_election.cc:288] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [CANDIDATE]: Term 1 election: Requested vote from peers 8febdecccbb54a4ebde27e121436ef09 (127.4.64.66:34141), 3b5c362788cc4a83b4b76426766a5e12 (127.4.64.67:37927)
I20260810 15:24:40.517791 26213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0c969f476b42158df083105e8d1ee0" candidate_uuid: "7e5f414f3e7248d184e6388ca94eedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8febdecccbb54a4ebde27e121436ef09"
I20260810 15:24:40.517822 26365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0c969f476b42158df083105e8d1ee0" candidate_uuid: "7e5f414f3e7248d184e6388ca94eedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5c362788cc4a83b4b76426766a5e12"
I20260810 15:24:40.517863 26213 raft_consensus.cc:3037] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:40.517892 26365 raft_consensus.cc:3037] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:40.518757 26213 raft_consensus.cc:2454] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5f414f3e7248d184e6388ca94eedec in term 1.
I20260810 15:24:40.518824 26365 raft_consensus.cc:2454] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5f414f3e7248d184e6388ca94eedec in term 1.
I20260810 15:24:40.518996 25993 leader_election.cc:302] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5c362788cc4a83b4b76426766a5e12, 7e5f414f3e7248d184e6388ca94eedec; no voters:
I20260810 15:24:40.519105 26442 raft_consensus.cc:2781] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:40.519276 26442 raft_consensus.cc:686] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 1 LEADER]: Becoming Leader. State: Replica: 7e5f414f3e7248d184e6388ca94eedec, State: Running, Role: LEADER
I20260810 15:24:40.519367 26442 consensus_queue.cc:237] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } }
W20260810 15:24:40.519871 26420 tablet.cc:2369] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:40.520243 25906 catalog_manager.cc:5495] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e5f414f3e7248d184e6388ca94eedec (127.4.64.65). New cstate: current_term: 1 leader_uuid: "7e5f414f3e7248d184e6388ca94eedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5f414f3e7248d184e6388ca94eedec" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 42481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 } health_report { overall_health: UNKNOWN } } }
I20260810 15:24:40.527899 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.525580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.4.64.65:42481)
0810 15:24:40.525645 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:40.525647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:40.525658 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:24:40.525698 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:40.525699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:40.525742 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:40.525815 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:24:40.525933 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.526417 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.527208 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:40.527406 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:40.527465 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:40.527598 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:24:40.527683 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:24:40.527785 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:40.527829 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:24:40.527840 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:24:40.529788 26365 raft_consensus.cc:1261] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 1 FOLLOWER]: Refusing update from remote peer 7e5f414f3e7248d184e6388ca94eedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:40.529924 26213 raft_consensus.cc:1261] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 1 FOLLOWER]: Refusing update from remote peer 7e5f414f3e7248d184e6388ca94eedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:40.530133 26442 consensus_queue.cc:1035] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5c362788cc4a83b4b76426766a5e12" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 37927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:40.530253 26448 consensus_queue.cc:1035] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8febdecccbb54a4ebde27e121436ef09" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:40.535671 25907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:40.535789 25907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:40.536646 25907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f43b951b07fe4553aaa33d2dff96135f: Sending DeleteTablet for 3 replicas of tablet ce0c969f476b42158df083105e8d1ee0
I20260810 15:24:40.537113 26192 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0c969f476b42158df083105e8d1ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:40 UTC) from {username='slave'} at 127.0.0.1:55500
I20260810 15:24:40.537149 26343 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0c969f476b42158df083105e8d1ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:40 UTC) from {username='slave'} at 127.0.0.1:53424
I20260810 15:24:40.537254 26046 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0c969f476b42158df083105e8d1ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:40 UTC) from {username='slave'} at 127.0.0.1:43292
I20260810 15:24:40.537415 26462 tablet_replica.cc:330] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: stopping tablet replica
I20260810 15:24:40.537482 26464 tablet_replica.cc:330] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec: stopping tablet replica
I20260810 15:24:40.537580 26464 raft_consensus.cc:2229] T ce0c969f476b42158df083105e8d1ee0 P 7e5f414f3e7248d184e6388ca94eedec [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:24:40.537724 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25977
I20260810 15:24:40.537597 26462 raft_consensus.cc:2229] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:40.542001 26462 raft_consensus.cc:2258] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:40.542313 26463 tablet_replica.cc:330] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: stopping tablet replica
I20260810 15:24:40.542448 26463 raft_consensus.cc:2229] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:40.542577 26463 raft_consensus.cc:2258] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:40.542692 26462 ts_tablet_manager.cc:1905] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:40.543020 26463 ts_tablet_manager.cc:1905] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:40.544564 26462 ts_tablet_manager.cc:1918] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:40.544654 26462 log.cc:1196] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/ce0c969f476b42158df083105e8d1ee0
I20260810 15:24:40.544726 26463 ts_tablet_manager.cc:1918] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:40.544811 26463 log.cc:1196] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/ce0c969f476b42158df083105e8d1ee0
I20260810 15:24:40.544967 26462 ts_tablet_manager.cc:1939] T ce0c969f476b42158df083105e8d1ee0 P 8febdecccbb54a4ebde27e121436ef09: Deleting consensus metadata
I20260810 15:24:40.545085 26463 ts_tablet_manager.cc:1939] T ce0c969f476b42158df083105e8d1ee0 P 3b5c362788cc4a83b4b76426766a5e12: Deleting consensus metadata
I20260810 15:24:40.545584 25890 catalog_manager.cc:4841] TS 8febdecccbb54a4ebde27e121436ef09 (127.4.64.66:34141): tablet ce0c969f476b42158df083105e8d1ee0 (table test-table [id=2dd1fa3d1a044ddea9d803999868c0f9]) successfully deleted
I20260810 15:24:40.545763 25890 catalog_manager.cc:4841] TS 3b5c362788cc4a83b4b76426766a5e12 (127.4.64.67:37927): tablet ce0c969f476b42158df083105e8d1ee0 (table test-table [id=2dd1fa3d1a044ddea9d803999868c0f9]) successfully deleted
W20260810 15:24:40.546109 25890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:42481 (error 108)
I20260810 15:24:40.546321 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 26121
I20260810 15:24:40.546608 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:40.546484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:42481 (local address 127.0.0.1:43304)
0810 15:24:40.546533 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:40.546570 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:24:40.546726 25890 catalog_manager.cc:4584] TS 7e5f414f3e7248d184e6388ca94eedec (127.4.64.65:42481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce0c969f476b42158df083105e8d1ee0: Network error: Client connection negotiation failed: client connection to 127.4.64.65:42481: connect: Connection refused (error 111)
I20260810 15:24:40.554988 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 26269
I20260810 15:24:40.563659 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 25871
2026-08-10T15:24:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:40 dist-test-slave-9zdj krb5kdc[26468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:40 dist-test-slave-9zdj krb5kdc[26468](info): set up 2 sockets
Aug 10 15:24:40 dist-test-slave-9zdj krb5kdc[26468](info): commencing operation
krb5kdc: starting...
W20260810 15:24:41.796097 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:41 dist-test-slave-9zdj krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:24:41.939308 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35987
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:42.024595 26485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:42.024891 26485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:42.024930 26485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:42.026145 26485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:42.026232 26485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:42.026254 26485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:42.026292 26485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:42.026321 26485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:42.027958 26485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:42.028266 26485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:42.028525 26485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:42.033864 26492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:42.034006 26485 server_base.cc:1034] running on GCE node
W20260810 15:24:42.033897 26491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.034075 26494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:42.034464 26485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:42.034797 26485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:42.035961 26485 hybrid_clock.cc:648] HybridClock initialized: now 1786375482035941 us; error 41 us; skew 500 ppm
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:42.038882 26485 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:24:42.039443 26485 webserver.cc:460] Webserver started at http://127.4.64.126:32973/ using document root <none> and password file <none>
I20260810 15:24:42.039666 26485 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:42.039708 26485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:42.039808 26485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:42.040853 26485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "b2df90207d9e4f5c99d14d2e6276ba5f"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "fd3d6a5c79790e703f08e8327bef332e"
server_key_iv: "46436c2f3f288616297f34ef8e1a30d6"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.041175 26485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "b2df90207d9e4f5c99d14d2e6276ba5f"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "fd3d6a5c79790e703f08e8327bef332e"
server_key_iv: "46436c2f3f288616297f34ef8e1a30d6"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.042644 26485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:42.043428 26503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.043653 26485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:42.043747 26485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "b2df90207d9e4f5c99d14d2e6276ba5f"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "fd3d6a5c79790e703f08e8327bef332e"
server_key_iv: "46436c2f3f288616297f34ef8e1a30d6"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.043812 26485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:42.058414 26485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:42.060781 26485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:42.060913 26485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:42.065333 26485 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:35987
I20260810 15:24:42.065457 26561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:35987 every 8 connection(s)
I20260810 15:24:42.065858 26485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:42.066854 26562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:42.069149 26562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f: Bootstrap starting.
I20260810 15:24:42.069799 26562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:42.070053 26562 log.cc:824] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:42.070783 26562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f: No bootstrap required, opened a new log
I20260810 15:24:42.072414 26562 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35987 } }
I20260810 15:24:42.072559 26562 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:42.072582 26562 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2df90207d9e4f5c99d14d2e6276ba5f, State: Initialized, Role: FOLLOWER
I20260810 15:24:42.072745 26562 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35987 } }
I20260810 15:24:42.072844 26562 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:42.072868 26562 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:42.072902 26562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:42.073411 26562 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35987 } }
I20260810 15:24:42.073498 26562 leader_election.cc:302] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2df90207d9e4f5c99d14d2e6276ba5f; no voters:
I20260810 15:24:42.073698 26562 leader_election.cc:288] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:42.073737 26565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:42.073865 26565 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [term 1 LEADER]: Becoming Leader. State: Replica: b2df90207d9e4f5c99d14d2e6276ba5f, State: Running, Role: LEADER
I20260810 15:24:42.073994 26562 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:42.074015 26565 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35987 } }
I20260810 15:24:42.074512 26567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35987 } } }
I20260810 15:24:42.074697 26567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:42.074713 26566 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2df90207d9e4f5c99d14d2e6276ba5f. Latest consensus state: current_term: 1 leader_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2df90207d9e4f5c99d14d2e6276ba5f" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35987 } } }
I20260810 15:24:42.074836 26566 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:42.075166 26578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:42.075837 26578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:42.075881 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 26485
I20260810 15:24:42.076017 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:42.076253 4353 external_mini_cluster.cc:1428] Setting key d71740765353245a1522c21851c51904
I20260810 15:24:42.077927 26578 catalog_manager.cc:1347] Generated new cluster ID: b337fdc426884e5a824bd5fcc5ff9331
I20260810 15:24:42.077988 26578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:42.096547 26578 catalog_manager.cc:1370] Generated new certificate authority record
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:42.097442 26578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:42.101904 26578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f: Generated new TSK 0
I20260810 15:24:42.102092 26578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:42.102890 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.076780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.4.64.126:35987)
0810 15:24:42.076992 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.077002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.077313 (+ 311us) server_negotiation.cc:407] Connection header received
0810 15:24:42.077511 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.077518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.077578 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.077757 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:42.090555 (+ 12798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.092150 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.093462 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.094187 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.094328 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.097778 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:42.097784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:42.097784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:42.097799 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:42.100720 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.101210 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.101215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.101215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.101349 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.101683 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.101684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.101684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.102077 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:42.102259 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.102402 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.102655 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.4.64.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
Entry for principal kudu/127.4.64.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab.
I20260810 15:24:42.137727 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35987
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:42.225273 26595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:42.225544 26595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:42.225584 26595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:42.226758 26595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:42.226841 26595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:42.226886 26595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:42.228559 26595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.65.keytab
--principal=kudu/127.4.64.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:42.228852 26595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:42.229118 26595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:42.234666 26600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.234680 26601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.234658 26603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:42.235239 26595 server_base.cc:1034] running on GCE node
I20260810 15:24:42.235427 26595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:42.235641 26595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:42.236820 26595 hybrid_clock.cc:648] HybridClock initialized: now 1786375482236790 us; error 43 us; skew 500 ppm
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:42.239708 26595 init.cc:377] Logged in from keytab as kudu/127.4.64.65@KRBTEST.COM (short username kudu)
I20260810 15:24:42.240190 26595 webserver.cc:460] Webserver started at http://127.4.64.65:43875/ using document root <none> and password file <none>
I20260810 15:24:42.240430 26595 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:42.240478 26595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:42.240624 26595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:42.241818 26595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "6d8a476c5cfc45b7ac57216ecd8136ca"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "6c2e47bb9ac53afe3e28bb6c1dfa07f6"
server_key_iv: "07c717a5fcdcc6384c65eb3948bb439d"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.242199 26595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "6d8a476c5cfc45b7ac57216ecd8136ca"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "6c2e47bb9ac53afe3e28bb6c1dfa07f6"
server_key_iv: "07c717a5fcdcc6384c65eb3948bb439d"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.243657 26595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:24:42.244509 26610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.244720 26595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:42.244832 26595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "6d8a476c5cfc45b7ac57216ecd8136ca"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "6c2e47bb9ac53afe3e28bb6c1dfa07f6"
server_key_iv: "07c717a5fcdcc6384c65eb3948bb439d"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.244910 26595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:42.253816 26595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:42.256268 26595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:42.256485 26595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:42.256785 26595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:42.257184 26595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:42.257231 26595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.257301 26595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:42.257332 26595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.264528 26595 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:33447
I20260810 15:24:42.264618 26740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:33447 every 8 connection(s)
I20260810 15:24:42.265012 26595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.65@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:42.274680 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 26595
I20260810 15:24:42.274834 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:24:42.275120 4353 external_mini_cluster.cc:1428] Setting key 46046d91b0ef10d41402914637d02ddc
I20260810 15:24:42.276773 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.266007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:40647 (local address 127.4.64.126:35987)
0810 15:24:42.266098 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.266101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.266374 (+ 273us) server_negotiation.cc:407] Connection header received
0810 15:24:42.266879 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.266880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.266893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.266957 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:42.268080 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.268654 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.269826 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.270067 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.270133 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.272707 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:42.272709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:42.272709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:42.272716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:42.274422 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.275158 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.275159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.275159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.275237 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.275884 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.275884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.275885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.276072 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:42.276152 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.276574 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.276671 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 15:24:42.277115 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.265967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35987 (local address 127.4.64.65:40647)
0810 15:24:42.266213 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.266269 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.266768 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.267066 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.267071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.267739 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:42.267965 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.267970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.268764 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.268765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.269733 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.269737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.270170 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.270171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.270298 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.270916 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:42.270926 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:42.272596 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:42.274583 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:42.274633 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:42.274636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:42.275069 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:42.275344 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:42.275345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:42.275345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:42.275526 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:42.276165 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:42.276174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:42.276395 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.276536 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.276785 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":128,"threads_started":1}
I20260810 15:24:42.277763 26742 heartbeater.cc:347] Connected to a master server at 127.4.64.126:35987
I20260810 15:24:42.277901 26742 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:42.278103 26742 heartbeater.cc:511] Master 127.4.64.126:35987 requested a full tablet report, sending...
I20260810 15:24:42.278770 26522 ts_manager.cc:194] Registered new tserver with Master: 6d8a476c5cfc45b7ac57216ecd8136ca (127.4.64.65:33447)
I20260810 15:24:42.279466 26522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.65@KRBTEST.COM'} at 127.4.64.65:40647
WARNING: no policy specified for kudu/127.4.64.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
Entry for principal kudu/127.4.64.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab.
I20260810 15:24:42.309253 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35987
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:42.398185 26747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:42.398412 26747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:42.398437 26747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:42.399550 26747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:42.399588 26747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:42.399614 26747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:42.401026 26747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.66.keytab
--principal=kudu/127.4.64.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:42.401233 26747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:42.401425 26747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:42.406816 26755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.406911 26752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.406816 26753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:42.407044 26747 server_base.cc:1034] running on GCE node
I20260810 15:24:42.407444 26747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:42.407678 26747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:42.408876 26747 hybrid_clock.cc:648] HybridClock initialized: now 1786375482408857 us; error 32 us; skew 500 ppm
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:42.411576 26747 init.cc:377] Logged in from keytab as kudu/127.4.64.66@KRBTEST.COM (short username kudu)
I20260810 15:24:42.412114 26747 webserver.cc:460] Webserver started at http://127.4.64.66:41823/ using document root <none> and password file <none>
I20260810 15:24:42.412392 26747 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:42.412477 26747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:42.412638 26747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:42.413805 26747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "64403119b656412baf2bf4ed3dbabbb9"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "77865e9e9e540aa8bd4484d287965217"
server_key_iv: "6ff70cdae519ba19099f65b56d83b6d3"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.414218 26747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "64403119b656412baf2bf4ed3dbabbb9"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "77865e9e9e540aa8bd4484d287965217"
server_key_iv: "6ff70cdae519ba19099f65b56d83b6d3"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.415634 26747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:24:42.416483 26763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.416682 26747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:42.416781 26747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "64403119b656412baf2bf4ed3dbabbb9"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "77865e9e9e540aa8bd4484d287965217"
server_key_iv: "6ff70cdae519ba19099f65b56d83b6d3"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.416882 26747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:42.430763 26747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:42.432941 26747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:42.433116 26747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:42.433378 26747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:42.433780 26747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:42.433845 26747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.433885 26747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:42.433913 26747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.440423 26747 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:41187
I20260810 15:24:42.440466 26891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:41187 every 8 connection(s)
I20260810 15:24:42.440889 26747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:42.446679 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 26747
I20260810 15:24:42.446842 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:24:42.447078 4353 external_mini_cluster.cc:1428] Setting key 5dac74b4b47e2082976eaef8adbc783d
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.66@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:42.452553 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.441878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:48389 (local address 127.4.64.126:35987)
0810 15:24:42.441975 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.441977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.442187 (+ 210us) server_negotiation.cc:407] Connection header received
0810 15:24:42.442766 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.442768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.442782 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.442845 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:42.443916 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.444467 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.445563 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.445836 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.445932 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.448601 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:42.448602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:42.448603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:42.448611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:42.450664 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.451379 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.451380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.451380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.451446 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.451795 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.451795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.451796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.451911 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:42.451966 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.452406 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.452483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:24:42.452975 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.441853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35987 (local address 127.4.64.66:48389)
0810 15:24:42.442045 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.442091 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.442643 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.442919 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.442923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.443569 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:42.443795 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.443801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.444584 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.444585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.445455 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.445457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.445946 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.445947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.446109 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.446659 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:42.446671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:42.448500 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:42.450831 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:42.450841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:42.450844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:42.451265 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:42.451532 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:42.451533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:42.451533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:42.451718 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:42.451983 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:42.451991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:42.452247 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.452388 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.452615 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260810 15:24:42.453703 26892 heartbeater.cc:347] Connected to a master server at 127.4.64.126:35987
I20260810 15:24:42.453828 26892 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:42.454064 26892 heartbeater.cc:511] Master 127.4.64.126:35987 requested a full tablet report, sending...
I20260810 15:24:42.454542 26522 ts_manager.cc:194] Registered new tserver with Master: 64403119b656412baf2bf4ed3dbabbb9 (127.4.64.66:41187)
I20260810 15:24:42.454962 26522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.66@KRBTEST.COM'} at 127.4.64.66:48389
WARNING: no policy specified for kudu/127.4.64.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
Entry for principal kudu/127.4.64.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab.
I20260810 15:24:42.482081 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35987
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:42.569591 26897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:42.569900 26897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:42.569926 26897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:42.571127 26897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:42.571182 26897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:42.571210 26897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:42.572705 26897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.67.keytab
--principal=kudu/127.4.64.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:35987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:42.572973 26897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:42.573225 26897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:42.579344 26902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.579339 26905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.579402 26903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:42.579782 26897 server_base.cc:1034] running on GCE node
I20260810 15:24:42.580049 26897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:42.580437 26897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:42.581666 26897 hybrid_clock.cc:648] HybridClock initialized: now 1786375482581638 us; error 46 us; skew 500 ppm
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:42.584619 26897 init.cc:377] Logged in from keytab as kudu/127.4.64.67@KRBTEST.COM (short username kudu)
I20260810 15:24:42.585151 26897 webserver.cc:460] Webserver started at http://127.4.64.67:35867/ using document root <none> and password file <none>
I20260810 15:24:42.585455 26897 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:42.585541 26897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:42.585744 26897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:42.587003 26897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "3811c05402ab447ba3a0e6a7af3ef84a"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "e87a2e4912ee4d0ac91e7934d25f4c31"
server_key_iv: "cb9e7ad7ce3adf5b002f3ddfd4c97d53"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.587435 26897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "3811c05402ab447ba3a0e6a7af3ef84a"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "e87a2e4912ee4d0ac91e7934d25f4c31"
server_key_iv: "cb9e7ad7ce3adf5b002f3ddfd4c97d53"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.588965 26897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:24:42.589988 26912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.590247 26897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:42.590368 26897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "3811c05402ab447ba3a0e6a7af3ef84a"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "e87a2e4912ee4d0ac91e7934d25f4c31"
server_key_iv: "cb9e7ad7ce3adf5b002f3ddfd4c97d53"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.590463 26897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:42.622473 26897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:42.624969 26897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:42.625185 26897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:42.625478 26897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:42.625900 26897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:42.625967 26897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.626015 26897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:42.626047 26897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.632922 26897 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:43711
I20260810 15:24:42.632988 27043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:43711 every 8 connection(s)
I20260810 15:24:42.633378 26897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:42.638798 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 26897
I20260810 15:24:42.638916 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:24:42.639127 4353 external_mini_cluster.cc:1428] Setting key c250046338c46720e334531ef875661b
Aug 10 15:24:42 dist-test-slave-9zdj krb5kdc[26468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375482, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.67@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:24:42.644304 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.634339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:60279 (local address 127.4.64.126:35987)
0810 15:24:42.634425 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.634428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.634623 (+ 195us) server_negotiation.cc:407] Connection header received
0810 15:24:42.635156 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.635157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.635171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.635221 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:42.636310 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.636839 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.637934 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.638141 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.638219 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.641065 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:42.641066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:42.641067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:42.641074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:42.642778 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.643267 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.643268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.643268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.643335 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.643597 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.643598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.643599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.643776 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:42.643844 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.644146 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.644223 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:24:42.644495 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.634252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:35987 (local address 127.4.64.67:60279)
0810 15:24:42.634499 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.634538 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.635044 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.635353 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.635356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.635997 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:42.636205 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.636210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.636945 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.636946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.637844 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.637846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.638241 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.638242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.638361 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.638909 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:24:42.638921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:42.640967 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:42.642862 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:42.642869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:42.642871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:42.643095 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:42.643407 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:24:42.643407 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:24:42.643408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:24:42.643511 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:24:42.643847 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:42.643853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:42.643983 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.644098 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.644253 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260810 15:24:42.644994 27044 heartbeater.cc:347] Connected to a master server at 127.4.64.126:35987
I20260810 15:24:42.645107 27044 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:42.645325 27044 heartbeater.cc:511] Master 127.4.64.126:35987 requested a full tablet report, sending...
I20260810 15:24:42.645855 26522 ts_manager.cc:194] Registered new tserver with Master: 3811c05402ab447ba3a0e6a7af3ef84a (127.4.64.67:43711)
I20260810 15:24:42.646293 26522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.64.67@KRBTEST.COM'} at 127.4.64.67:60279
I20260810 15:24:42.651266 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:24:42.660116 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.652972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.4.64.126:35987)
0810 15:24:42.653018 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.653020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.653135 (+ 115us) server_negotiation.cc:407] Connection header received
0810 15:24:42.653196 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.653197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.653209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.653329 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:42.654074 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.654605 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.655453 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.655658 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.655718 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.657049 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:42.657050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:42.657051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:42.657057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:42.658718 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.659133 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.659134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.659135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.659300 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:42.659572 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:42.659574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:42.659574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:42.659755 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:42.659852 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.659972 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.660044 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:24:42.662211 26522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:24:42.664008 26522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:24:42.671814 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.667650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.4.64.66:41187)
0810 15:24:42.667875 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.667881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.668072 (+ 191us) server_negotiation.cc:407] Connection header received
0810 15:24:42.668127 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.668130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.668145 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.668238 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:42.668414 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.669592 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.670745 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.671300 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.671403 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.671476 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.671652 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.671709 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260810 15:24:42.671793 27063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.667370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:41187 (local address 127.0.0.1:59472)
0810 15:24:42.667933 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.667944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.667996 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.668259 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.668260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.668263 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:42.668309 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.668311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.669746 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.669747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.670619 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.670622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.671403 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.671404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.671500 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.671516 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.671632 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.671702 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":109,"threads_started":1}
I20260810 15:24:42.672102 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.667021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:33447 (local address 127.0.0.1:41954)
0810 15:24:42.667364 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.667413 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.667498 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.667892 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.667896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.667898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:42.667977 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.667980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.669111 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.669111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.670636 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.670639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.671780 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.671781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.671853 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.671857 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.671950 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.672016 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":5,"thread_start_us":233,"threads_started":1}
I20260810 15:24:42.672360 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.667025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:43711 (local address 127.0.0.1:34580)
0810 15:24:42.667436 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.667449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.667694 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.667934 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.667935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.667937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:42.667980 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.667981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.669467 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.669468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.670715 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.670718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.672105 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.672106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.672187 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.672191 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.672251 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.672300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":67,"threads_started":1}
I20260810 15:24:42.672446 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.667141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.4.64.65:33447)
0810 15:24:42.667371 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.667377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.667601 (+ 224us) server_negotiation.cc:407] Connection header received
0810 15:24:42.667681 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.667683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.667700 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.667786 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:42.668069 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.668983 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.670766 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.671605 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.671678 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.671738 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.672287 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.672349 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260810 15:24:42.672964 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.667348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.4.64.67:43711)
0810 15:24:42.667582 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.667587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.667599 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:24:42.667829 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.667832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.667851 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.667942 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:42.668036 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.669332 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.670812 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.671590 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.671683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.671738 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.672441 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.672878 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
I20260810 15:24:42.673318 26818 tablet_service.cc:1467] Processing CreateTablet for tablet 73a495bf50d7415d91a532b3ae775a48 (DEFAULT_TABLE table=test-table [id=8b0355b2dbf34e82835a1f06a53c1517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:42.673336 26664 tablet_service.cc:1467] Processing CreateTablet for tablet 73a495bf50d7415d91a532b3ae775a48 (DEFAULT_TABLE table=test-table [id=8b0355b2dbf34e82835a1f06a53c1517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:42.673594 26968 tablet_service.cc:1467] Processing CreateTablet for tablet 73a495bf50d7415d91a532b3ae775a48 (DEFAULT_TABLE table=test-table [id=8b0355b2dbf34e82835a1f06a53c1517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:42.673799 26818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a495bf50d7415d91a532b3ae775a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:42.673846 26664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a495bf50d7415d91a532b3ae775a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:42.673995 26968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a495bf50d7415d91a532b3ae775a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:42.677117 27065 tablet_bootstrap.cc:492] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: Bootstrap starting.
I20260810 15:24:42.677752 27067 tablet_bootstrap.cc:492] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Bootstrap starting.
I20260810 15:24:42.678049 27065 tablet_bootstrap.cc:654] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:42.678567 27067 tablet_bootstrap.cc:654] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:42.678813 27065 log.cc:824] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:42.679239 27067 log.cc:824] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:42.679600 27066 tablet_bootstrap.cc:492] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: Bootstrap starting.
I20260810 15:24:42.679808 27065 tablet_bootstrap.cc:492] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: No bootstrap required, opened a new log
I20260810 15:24:42.679896 27065 ts_tablet_manager.cc:1397] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:24:42.680018 27067 tablet_bootstrap.cc:492] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: No bootstrap required, opened a new log
I20260810 15:24:42.680114 27067 ts_tablet_manager.cc:1397] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:24:42.680361 27066 tablet_bootstrap.cc:654] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:42.681006 27066 log.cc:824] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:42.681950 27066 tablet_bootstrap.cc:492] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: No bootstrap required, opened a new log
I20260810 15:24:42.682034 27066 ts_tablet_manager.cc:1397] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:24:42.681977 27067 raft_consensus.cc:348] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.682205 27067 raft_consensus.cc:374] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:42.682260 27067 raft_consensus.cc:729] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3811c05402ab447ba3a0e6a7af3ef84a, State: Initialized, Role: FOLLOWER
I20260810 15:24:42.682389 27067 consensus_queue.cc:260] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.682430 27065 raft_consensus.cc:348] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.682608 27065 raft_consensus.cc:374] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:42.682642 27065 raft_consensus.cc:729] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8a476c5cfc45b7ac57216ecd8136ca, State: Initialized, Role: FOLLOWER
I20260810 15:24:42.682703 27067 ts_tablet_manager.cc:1428] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:24:42.682842 27044 heartbeater.cc:503] Master 127.4.64.126:35987 was elected leader, sending a full tablet report...
I20260810 15:24:42.683142 27065 consensus_queue.cc:260] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.683493 27065 ts_tablet_manager.cc:1428] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260810 15:24:42.683521 26742 heartbeater.cc:503] Master 127.4.64.126:35987 was elected leader, sending a full tablet report...
I20260810 15:24:42.684476 27066 raft_consensus.cc:348] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.684677 27066 raft_consensus.cc:374] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:42.684708 27066 raft_consensus.cc:729] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64403119b656412baf2bf4ed3dbabbb9, State: Initialized, Role: FOLLOWER
I20260810 15:24:42.684828 27066 consensus_queue.cc:260] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.685081 27066 ts_tablet_manager.cc:1428] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:24:42.685141 26892 heartbeater.cc:503] Master 127.4.64.126:35987 was elected leader, sending a full tablet report...
W20260810 15:24:42.691479 26893 tablet.cc:2369] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:42.734007 27071 raft_consensus.cc:482] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:42.734154 27071 raft_consensus.cc:504] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.734526 27071 leader_election.cc:288] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d8a476c5cfc45b7ac57216ecd8136ca (127.4.64.65:33447), 64403119b656412baf2bf4ed3dbabbb9 (127.4.64.66:41187)
I20260810 15:24:42.737522 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.734722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:41187 (local address 127.4.64.67:33163)
0810 15:24:42.734808 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.734820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.734869 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.735031 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.735032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.735034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:42.735119 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.735120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.735868 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.735869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.736670 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.736672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.737244 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.737245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.737331 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.737334 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.737396 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.737438 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":5}
I20260810 15:24:42.737515 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.734852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:33163 (local address 127.4.64.66:41187)
0810 15:24:42.734896 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.734898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.734908 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:42.734939 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.734940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.734955 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.735025 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:42.735288 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.735780 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.736767 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.737147 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.737235 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.737247 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.737414 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.737452 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:24:42.737728 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.734821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:59749 (local address 127.4.64.65:33447)
0810 15:24:42.734873 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.734876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.735097 (+ 221us) server_negotiation.cc:407] Connection header received
0810 15:24:42.735173 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.735175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.735186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.735256 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:42.735471 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.735951 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.736886 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.737294 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.737378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.737390 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.737546 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.737594 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:24:42.737927 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.734734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:33447 (local address 127.4.64.67:59749)
0810 15:24:42.735012 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.735022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:42.735104 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:42.735312 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:42.735313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:42.735315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:42.735390 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.735392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.736041 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.736042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.736788 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:42.736790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.737394 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.737396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:42.737468 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.737472 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:42.737540 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:24:42.737761 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":197,"threads_started":1}
I20260810 15:24:42.737922 26840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a495bf50d7415d91a532b3ae775a48" candidate_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64403119b656412baf2bf4ed3dbabbb9" is_pre_election: true
I20260810 15:24:42.738080 26840 raft_consensus.cc:2454] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3811c05402ab447ba3a0e6a7af3ef84a in term 0.
I20260810 15:24:42.738338 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a495bf50d7415d91a532b3ae775a48" candidate_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" is_pre_election: true
I20260810 15:24:42.738374 26913 leader_election.cc:302] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3811c05402ab447ba3a0e6a7af3ef84a, 64403119b656412baf2bf4ed3dbabbb9; no voters:
I20260810 15:24:42.738451 26688 raft_consensus.cc:2454] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3811c05402ab447ba3a0e6a7af3ef84a in term 0.
I20260810 15:24:42.738499 27071 raft_consensus.cc:2781] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:24:42.738555 27071 raft_consensus.cc:482] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:24:42.738606 27071 raft_consensus.cc:3037] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:42.739355 27071 raft_consensus.cc:504] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.739502 27071 leader_election.cc:288] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [CANDIDATE]: Term 1 election: Requested vote from peers 6d8a476c5cfc45b7ac57216ecd8136ca (127.4.64.65:33447), 64403119b656412baf2bf4ed3dbabbb9 (127.4.64.66:41187)
I20260810 15:24:42.739661 26688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a495bf50d7415d91a532b3ae775a48" candidate_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca"
I20260810 15:24:42.739670 26840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a495bf50d7415d91a532b3ae775a48" candidate_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64403119b656412baf2bf4ed3dbabbb9"
I20260810 15:24:42.739773 26688 raft_consensus.cc:3037] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:42.739768 26840 raft_consensus.cc:3037] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:42.740509 26840 raft_consensus.cc:2454] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3811c05402ab447ba3a0e6a7af3ef84a in term 1.
I20260810 15:24:42.740677 26688 raft_consensus.cc:2454] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3811c05402ab447ba3a0e6a7af3ef84a in term 1.
I20260810 15:24:42.740675 26913 leader_election.cc:302] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3811c05402ab447ba3a0e6a7af3ef84a, 64403119b656412baf2bf4ed3dbabbb9; no voters:
I20260810 15:24:42.740855 27071 raft_consensus.cc:2781] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:42.741034 27071 raft_consensus.cc:686] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 1 LEADER]: Becoming Leader. State: Replica: 3811c05402ab447ba3a0e6a7af3ef84a, State: Running, Role: LEADER
I20260810 15:24:42.741255 27071 consensus_queue.cc:237] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } }
I20260810 15:24:42.742271 26522 catalog_manager.cc:5495] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a reported cstate change: term changed from 0 to 1, leader changed from <none> to 3811c05402ab447ba3a0e6a7af3ef84a (127.4.64.67). New cstate: current_term: 1 leader_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3811c05402ab447ba3a0e6a7af3ef84a" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 43711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 } health_report { overall_health: UNKNOWN } } }
I20260810 15:24:42.764878 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.762203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.4.64.67:43711)
0810 15:24:42.762277 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.762279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.762290 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:24:42.762379 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.762379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.762400 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.762466 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:24:42.762678 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.763183 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.764051 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.764240 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.764296 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.764517 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:24:42.764621 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:24:42.764684 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.764756 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.764775 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 15:24:42.765864 26743 tablet.cc:2369] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:42.767021 26688 raft_consensus.cc:1261] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 1 FOLLOWER]: Refusing update from remote peer 3811c05402ab447ba3a0e6a7af3ef84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:42.767026 26840 raft_consensus.cc:1261] T 73a495bf50d7415d91a532b3ae775a48 P 64403119b656412baf2bf4ed3dbabbb9 [term 1 FOLLOWER]: Refusing update from remote peer 3811c05402ab447ba3a0e6a7af3ef84a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:42.767452 27076 consensus_queue.cc:1035] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d8a476c5cfc45b7ac57216ecd8136ca" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 33447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:42.767582 27071 consensus_queue.cc:1035] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64403119b656412baf2bf4ed3dbabbb9" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:42.770157 27079 mvcc.cc:205] Tried to move back new op lower bound from 7316993977410932736 to 7316993977308987392. Current Snapshot: MvccSnapshot[applied={T|T < 7316993977410932736}]
I20260810 15:24:42.770439 27080 mvcc.cc:205] Tried to move back new op lower bound from 7316993977410932736 to 7316993977308987392. Current Snapshot: MvccSnapshot[applied={T|T < 7316993977410932736}]
I20260810 15:24:42.770689 27081 mvcc.cc:205] Tried to move back new op lower bound from 7316993977410932736 to 7316993977308987392. Current Snapshot: MvccSnapshot[applied={T|T < 7316993977410932736}]
I20260810 15:24:42.772745 26521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:42.772858 26521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:42.773782 26521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2df90207d9e4f5c99d14d2e6276ba5f: Sending DeleteTablet for 3 replicas of tablet 73a495bf50d7415d91a532b3ae775a48
I20260810 15:24:42.774124 26664 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a495bf50d7415d91a532b3ae775a48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:42 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:41954
I20260810 15:24:42.774202 26968 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a495bf50d7415d91a532b3ae775a48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:42 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:34580
I20260810 15:24:42.774211 26818 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a495bf50d7415d91a532b3ae775a48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:42 UTC) from {username='kudu', principal='kudu/127.4.64.126@KRBTEST.COM'} at 127.0.0.1:59472
I20260810 15:24:42.774389 27091 tablet_replica.cc:330] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca: stopping tablet replica
I20260810 15:24:42.774483 27091 raft_consensus.cc:2229] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:42.774595 27091 raft_consensus.cc:2258] T 73a495bf50d7415d91a532b3ae775a48 P 6d8a476c5cfc45b7ac57216ecd8136ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:42.774784 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 26595
I20260810 15:24:42.774669 27093 tablet_replica.cc:330] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: stopping tablet replica
W20260810 15:24:42.791385 26507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:33447 (error 108)
I20260810 15:24:42.791683 27093 raft_consensus.cc:2229] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:24:42.791754 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 26747
I20260810 15:24:42.792006 27093 raft_consensus.cc:2258] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:42.794793 27093 ts_tablet_manager.cc:1905] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:42.795786 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.791907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:33447 (local address 127.0.0.1:41970)
0810 15:24:42.795666 (+ 3759us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.795736 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:33447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3725}
W20260810 15:24:42.797254 26507 catalog_manager.cc:4584] TS 6d8a476c5cfc45b7ac57216ecd8136ca (127.4.64.65:33447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73a495bf50d7415d91a532b3ae775a48: Network error: Client connection negotiation failed: client connection to 127.4.64.65:33447: connect: Connection refused (error 111)
I20260810 15:24:42.797859 27093 ts_tablet_manager.cc:1918] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:42.797948 27093 log.cc:1196] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/73a495bf50d7415d91a532b3ae775a48
I20260810 15:24:42.798270 27093 ts_tablet_manager.cc:1939] T 73a495bf50d7415d91a532b3ae775a48 P 3811c05402ab447ba3a0e6a7af3ef84a: Deleting consensus metadata
I20260810 15:24:42.798830 26504 catalog_manager.cc:4841] TS 3811c05402ab447ba3a0e6a7af3ef84a (127.4.64.67:43711): tablet 73a495bf50d7415d91a532b3ae775a48 (table test-table [id=8b0355b2dbf34e82835a1f06a53c1517]) successfully deleted
I20260810 15:24:42.802821 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 26897
I20260810 15:24:42.803013 27060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.802810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:41187 (local address 127.0.0.1:59486)
0810 15:24:42.802932 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:42.802973 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.66:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 15:24:42.810776 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 26485
2026-08-10T15:24:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2248 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T15:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:42Z Disabled control of system clock
I20260810 15:24:42.843834 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34201
--webserver_interface=127.4.64.126
--webserver_port=0
--builtin_ntp_servers=127.4.64.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:34201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:42.929237 27099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:42.929476 27099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:42.929497 27099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:42.930824 27099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:42.930876 27099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:42.930888 27099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:42.930902 27099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:42.930913 27099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:42.932307 27099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:34201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:42.932581 27099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:42.932911 27099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:42.938597 27105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.938584 27106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:42.938658 27108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:42.938948 27099 server_base.cc:1034] running on GCE node
I20260810 15:24:42.939196 27099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:42.939446 27099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:42.940702 27099 hybrid_clock.cc:648] HybridClock initialized: now 1786375482940680 us; error 29 us; skew 500 ppm
I20260810 15:24:42.942224 27099 webserver.cc:460] Webserver started at http://127.4.64.126:41621/ using document root <none> and password file <none>
I20260810 15:24:42.942503 27099 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:42.942554 27099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:42.942713 27099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:42.943840 27099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "8e3a921e8f0a493f81088f7cce9ceb7e"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "3eff27b9845a813ad57d1cbc021c1b8f"
server_key_iv: "f65da8bdb28a0092365569f375f37803"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.944274 27099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "8e3a921e8f0a493f81088f7cce9ceb7e"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "3eff27b9845a813ad57d1cbc021c1b8f"
server_key_iv: "f65da8bdb28a0092365569f375f37803"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.945701 27099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:24:42.946476 27114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:42.946663 27099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:42.946758 27099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "8e3a921e8f0a493f81088f7cce9ceb7e"
format_stamp: "Formatted at 2026-08-10 15:24:42 on dist-test-slave-9zdj"
server_key: "3eff27b9845a813ad57d1cbc021c1b8f"
server_key_iv: "f65da8bdb28a0092365569f375f37803"
server_key_version: "encryptionkey@0"
I20260810 15:24:42.946831 27099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:42.963644 27099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:42.964102 27099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:42.964308 27099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:42.969034 27099 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:34201
I20260810 15:24:42.969077 27174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:34201 every 8 connection(s)
I20260810 15:24:42.969458 27099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:42.970570 27175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:42.972846 27175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e: Bootstrap starting.
I20260810 15:24:42.973428 27175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:42.973704 27175 log.cc:824] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:42.974450 27175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e: No bootstrap required, opened a new log
I20260810 15:24:42.976004 27175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34201 } }
I20260810 15:24:42.976155 27175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:42.976181 27175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3a921e8f0a493f81088f7cce9ceb7e, State: Initialized, Role: FOLLOWER
I20260810 15:24:42.976298 27175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34201 } }
I20260810 15:24:42.976409 27175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:42.976449 27175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:42.976500 27175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:42.977057 27175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34201 } }
I20260810 15:24:42.977197 27175 leader_election.cc:302] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e3a921e8f0a493f81088f7cce9ceb7e; no voters:
I20260810 15:24:42.977391 27175 leader_election.cc:288] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:42.977458 27178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:42.977689 27175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:42.977701 27178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [term 1 LEADER]: Becoming Leader. State: Replica: 8e3a921e8f0a493f81088f7cce9ceb7e, State: Running, Role: LEADER
I20260810 15:24:42.977864 27178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34201 } }
I20260810 15:24:42.978300 27180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e3a921e8f0a493f81088f7cce9ceb7e. Latest consensus state: current_term: 1 leader_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34201 } } }
I20260810 15:24:42.978386 27179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3a921e8f0a493f81088f7cce9ceb7e" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 34201 } } }
I20260810 15:24:42.978404 27180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:42.978453 27179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:42.978657 27190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:42.979246 27190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:42.979293 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27099
I20260810 15:24:42.979390 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:42.979591 4353 external_mini_cluster.cc:1428] Setting key 14d50d93ae70ab10ff573696283631a5
I20260810 15:24:42.981052 27190 catalog_manager.cc:1347] Generated new cluster ID: 97293526067e4aa6916600f400caa5a6
I20260810 15:24:42.981110 27190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:42.986363 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:42.980089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.4.64.126:34201)
0810 15:24:42.980703 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:42.980712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:42.980856 (+ 144us) server_negotiation.cc:407] Connection header received
0810 15:24:42.980983 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:42.980990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:42.981061 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:42.981205 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:42.981572 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.982976 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.984108 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:42.984714 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:42.984833 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:42.985633 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:42.985637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:42.985638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:42.985846 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:42.985885 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:42.985899 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:42.985962 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:42.986072 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 15:24:42.986147 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":90,"threads_started":1}
I20260810 15:24:43.016764 27190 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:43.017510 27190 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:43.023321 27190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e: Generated new TSK 0
I20260810 15:24:43.023523 27190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:43.028455 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.65:0
--local_ip_for_outbound_sockets=127.4.64.65
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34201
--builtin_ntp_servers=127.4.64.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:43.118548 27205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:43.118798 27205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:43.118824 27205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:43.119967 27205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:43.120007 27205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:43.120031 27205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.65
I20260810 15:24:43.121331 27205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.64.65
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.65
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:43.121555 27205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:43.121869 27205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:43.127416 27212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:43.127427 27210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:43.127524 27205 server_base.cc:1034] running on GCE node
W20260810 15:24:43.127431 27214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:43.127940 27205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:43.128294 27205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:43.129482 27205 hybrid_clock.cc:648] HybridClock initialized: now 1786375483129460 us; error 39 us; skew 500 ppm
I20260810 15:24:43.130930 27205 webserver.cc:460] Webserver started at http://127.4.64.65:36479/ using document root <none> and password file <none>
I20260810 15:24:43.131141 27205 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:43.131249 27205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:43.131350 27205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:43.132433 27205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data/instance:
uuid: "b723fca91f334089b9103bc9b0a41022"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "e2400553c3e9af06e3c98208374d76d2"
server_key_iv: "a173049f16396ade16b76de2940df91c"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.132754 27205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance:
uuid: "b723fca91f334089b9103bc9b0a41022"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "e2400553c3e9af06e3c98208374d76d2"
server_key_iv: "a173049f16396ade16b76de2940df91c"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.134258 27205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:43.135134 27221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.135344 27205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:43.135442 27205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal
uuid: "b723fca91f334089b9103bc9b0a41022"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "e2400553c3e9af06e3c98208374d76d2"
server_key_iv: "a173049f16396ade16b76de2940df91c"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.135524 27205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:43.147357 27205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:43.147742 27205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:43.147912 27205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:43.148182 27205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:43.148554 27205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:43.148591 27205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.148614 27205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:43.148627 27205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.155301 27205 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.65:39521
I20260810 15:24:43.155397 27352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.65:39521 every 8 connection(s)
I20260810 15:24:43.155767 27205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/data/info.pb
I20260810 15:24:43.162366 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.156709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.65:56383 (local address 127.4.64.126:34201)
0810 15:24:43.156812 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.156816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.157063 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:24:43.157832 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.157834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.157850 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.157902 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:43.158841 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.159423 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.160608 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.160854 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.160947 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.162018 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:43.162019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:43.162020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:43.162025 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:43.162042 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:43.162047 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:43.162102 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.162238 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.162274 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:24:43.162621 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.156656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34201 (local address 127.4.64.65:56383)
0810 15:24:43.156903 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.156953 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.157660 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.157996 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.158001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.158012 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:43.158721 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.158728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.159548 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.159550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.160491 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.160494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.160958 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.160959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.161116 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.161832 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:43.161833 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:43.161866 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:43.161892 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:43.162117 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:43.162123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:43.162126 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.162237 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.162363 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260810 15:24:43.163304 27353 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34201
I20260810 15:24:43.163452 27353 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:43.163702 27353 heartbeater.cc:511] Master 127.4.64.126:34201 requested a full tablet report, sending...
I20260810 15:24:43.164338 27134 ts_manager.cc:194] Registered new tserver with Master: b723fca91f334089b9103bc9b0a41022 (127.4.64.65:39521)
I20260810 15:24:43.164731 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27205
I20260810 15:24:43.164845 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-0/wal/instance
I20260810 15:24:43.165073 4353 external_mini_cluster.cc:1428] Setting key c86a2f79e9c3852cc9e3a8221d675cf8
I20260810 15:24:43.165102 27134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.65:56383
I20260810 15:24:43.166289 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.66:0
--local_ip_for_outbound_sockets=127.4.64.66
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34201
--builtin_ntp_servers=127.4.64.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:24:43.253715 27356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:43.253947 27356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:43.253973 27356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:43.255199 27356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:43.255247 27356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:43.255275 27356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.66
I20260810 15:24:43.256699 27356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.64.66
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.66
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:43.256987 27356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:43.257325 27356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:43.262933 27366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:43.262923 27364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:43.262918 27363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:43.263168 27356 server_base.cc:1034] running on GCE node
I20260810 15:24:43.263597 27356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:43.263871 27356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:43.265029 27356 hybrid_clock.cc:648] HybridClock initialized: now 1786375483265003 us; error 44 us; skew 500 ppm
I20260810 15:24:43.266505 27356 webserver.cc:460] Webserver started at http://127.4.64.66:36075/ using document root <none> and password file <none>
I20260810 15:24:43.266997 27356 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:43.267066 27356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:43.267346 27356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:43.268513 27356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data/instance:
uuid: "122d863e36724f36a274eb7476329df0"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "e91616ab80a402db74a4c52d2d731240"
server_key_iv: "89c1a4b32128350dfca8e3b132100343"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.268893 27356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance:
uuid: "122d863e36724f36a274eb7476329df0"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "e91616ab80a402db74a4c52d2d731240"
server_key_iv: "89c1a4b32128350dfca8e3b132100343"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.270498 27356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260810 15:24:43.271485 27373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.271764 27356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:43.271893 27356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal
uuid: "122d863e36724f36a274eb7476329df0"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "e91616ab80a402db74a4c52d2d731240"
server_key_iv: "89c1a4b32128350dfca8e3b132100343"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.271976 27356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:43.288160 27356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:43.288585 27356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:43.288763 27356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:43.289032 27356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:43.289485 27356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:43.289547 27356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.289585 27356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:43.289690 27356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.296036 27356 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.66:43509
I20260810 15:24:43.296113 27500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.66:43509 every 8 connection(s)
I20260810 15:24:43.296496 27356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/data/info.pb
I20260810 15:24:43.302801 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.297390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.66:39243 (local address 127.4.64.126:34201)
0810 15:24:43.297445 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.297448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.297842 (+ 394us) server_negotiation.cc:407] Connection header received
0810 15:24:43.298757 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.298759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.298771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.298836 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:43.299552 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.300071 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.301116 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.301322 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.301410 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.302451 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:43.302452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:43.302453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:43.302459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:43.302475 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:43.302479 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:43.302549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.302659 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.302688 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:24:43.302982 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.297404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34201 (local address 127.4.64.66:39243)
0810 15:24:43.297685 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.297740 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.298636 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.298909 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.298914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.298924 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:43.299445 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.299450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.300178 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.300179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.300997 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.300999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.301418 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.301419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.301568 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.302180 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:43.302181 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:43.302293 (+ 112us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:43.302317 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:43.302564 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:43.302570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:43.302573 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.302640 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.302753 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20260810 15:24:43.303128 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27356
I20260810 15:24:43.303324 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal/instance
I20260810 15:24:43.303573 4353 external_mini_cluster.cc:1428] Setting key c33c3c81aa8e28f15e8eef070759386a
I20260810 15:24:43.303608 27501 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34201
I20260810 15:24:43.303733 27501 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:43.303946 27501 heartbeater.cc:511] Master 127.4.64.126:34201 requested a full tablet report, sending...
I20260810 15:24:43.304390 27134 ts_manager.cc:194] Registered new tserver with Master: 122d863e36724f36a274eb7476329df0 (127.4.64.66:43509)
I20260810 15:24:43.304636 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.64.67:0
--local_ip_for_outbound_sockets=127.4.64.67
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34201
--builtin_ntp_servers=127.4.64.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:24:43.304855 27134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.66:39243
W20260810 15:24:43.390945 27504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:43.391162 27504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:43.391187 27504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:43.392288 27504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:43.392325 27504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:43.392351 27504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.64.67
I20260810 15:24:43.393700 27504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.64.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.64.67
--webserver_port=0
--tserver_master_addrs=127.4.64.126:34201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.64.67
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:43.393903 27504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:43.394163 27504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:43.399559 27512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:43.399621 27509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:43.399566 27510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:43.399887 27504 server_base.cc:1034] running on GCE node
I20260810 15:24:43.400089 27504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:43.400429 27504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:43.401700 27504 hybrid_clock.cc:648] HybridClock initialized: now 1786375483401667 us; error 36 us; skew 500 ppm
I20260810 15:24:43.403067 27504 webserver.cc:460] Webserver started at http://127.4.64.67:45847/ using document root <none> and password file <none>
I20260810 15:24:43.403399 27504 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:43.403481 27504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:43.403638 27504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:43.404772 27504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data/instance:
uuid: "f01260fcddb44777a2ed327a1d1f12d1"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "4eaa45025f3e03f6a504d94f21bff9a0"
server_key_iv: "11d62742cd91f38ecddd1df5bfd7a1c3"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.405180 27504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance:
uuid: "f01260fcddb44777a2ed327a1d1f12d1"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "4eaa45025f3e03f6a504d94f21bff9a0"
server_key_iv: "11d62742cd91f38ecddd1df5bfd7a1c3"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.406679 27504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:43.407538 27520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.407753 27504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:43.407862 27504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal
uuid: "f01260fcddb44777a2ed327a1d1f12d1"
format_stamp: "Formatted at 2026-08-10 15:24:43 on dist-test-slave-9zdj"
server_key: "4eaa45025f3e03f6a504d94f21bff9a0"
server_key_iv: "11d62742cd91f38ecddd1df5bfd7a1c3"
server_key_version: "encryptionkey@0"
I20260810 15:24:43.407939 27504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:43.423307 27504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:43.423692 27504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:43.423861 27504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:43.424108 27504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:24:43.424481 27504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:24:43.424538 27504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.424582 27504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:24:43.424609 27504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:43.430786 27504 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.67:35523
I20260810 15:24:43.430825 27645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.67:35523 every 8 connection(s)
I20260810 15:24:43.431268 27504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/data/info.pb
I20260810 15:24:43.437076 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.432070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54135 (local address 127.4.64.126:34201)
0810 15:24:43.432114 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.432117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.432349 (+ 232us) server_negotiation.cc:407] Connection header received
0810 15:24:43.432898 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.432899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.432913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.432981 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:43.433878 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.434397 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.435506 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.435718 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.435805 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.436765 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:43.436766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:43.436766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:43.436772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:43.436792 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:43.436796 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:43.436868 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.436953 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.436989 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:24:43.437321 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.432029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:34201 (local address 127.4.64.67:54135)
0810 15:24:43.432218 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.432255 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.432787 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.433052 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.433056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.433066 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:24:43.433775 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.433782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.434507 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.434508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.435405 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.435408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.435812 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.435813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.435964 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.436577 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:24:43.436584 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:24:43.436614 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:24:43.436637 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:24:43.436869 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:24:43.436877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:24:43.436880 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.436947 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.437058 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260810 15:24:43.437914 27646 heartbeater.cc:347] Connected to a master server at 127.4.64.126:34201
I20260810 15:24:43.438006 27646 heartbeater.cc:464] Registering TS with master...
I20260810 15:24:43.438217 27646 heartbeater.cc:511] Master 127.4.64.126:34201 requested a full tablet report, sending...
I20260810 15:24:43.438666 27134 ts_manager.cc:194] Registered new tserver with Master: f01260fcddb44777a2ed327a1d1f12d1 (127.4.64.67:35523)
I20260810 15:24:43.439093 27134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.64.67:54135
I20260810 15:24:43.441103 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27504
I20260810 15:24:43.441226 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal/instance
I20260810 15:24:43.441468 4353 external_mini_cluster.cc:1428] Setting key 64806f28751429dc8f2ef3650b95d38a
I20260810 15:24:43.442541 4353 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:24:43.447691 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.444268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.4.64.126:34201)
0810 15:24:43.444314 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.444316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.444439 (+ 123us) server_negotiation.cc:407] Connection header received
0810 15:24:43.444479 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.444480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.444491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.444549 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:43.444733 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.445228 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.446161 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.446477 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.446541 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.447373 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:43.447374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:43.447374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:43.447379 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:43.447393 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:43.447397 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:43.447450 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.447587 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.447617 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:24:43.449752 27134 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:24:43.451529 27134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:24:43.458465 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.454393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.67:35523 (local address 127.0.0.1:57260)
0810 15:24:43.454683 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.454692 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.454736 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.455147 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.455148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.455150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:43.455191 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.455192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.456103 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.456105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.457099 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.457101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.458115 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.458116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.458200 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.458216 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.458299 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.458350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":71,"threads_started":1}
I20260810 15:24:43.458473 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.454529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.4.64.67:35523)
0810 15:24:43.454894 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.454899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.454908 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:24:43.454961 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.454962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.454997 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.455077 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:43.455302 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.455999 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.457196 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.458018 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.458107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.458227 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.458317 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.458360 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":104,"threads_started":1}
I20260810 15:24:43.458635 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.454192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.4.64.65:39521)
0810 15:24:43.454393 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.454399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.454628 (+ 229us) server_negotiation.cc:407] Connection header received
0810 15:24:43.454906 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.454909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.454925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.455003 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:43.455169 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.455888 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.457375 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.458144 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.458207 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.458288 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.458493 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.458543 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260810 15:24:43.458951 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.454043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:39521 (local address 127.0.0.1:40730)
0810 15:24:43.454507 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.454552 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.454794 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.455015 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.455016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.455019 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:43.455094 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.455096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.455983 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.455984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.457291 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.457293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.458326 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.458327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.458392 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.458396 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.458793 (+ 397us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.458860 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":5,"thread_start_us":157,"threads_started":1}
I20260810 15:24:43.459583 27575 tablet_service.cc:1467] Processing CreateTablet for tablet 90f44b95fc8f436bbbc576836655c23f (DEFAULT_TABLE table=test-table [id=815d1e3da4a748eeb827ae04f281e06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:43.459869 27278 tablet_service.cc:1467] Processing CreateTablet for tablet 90f44b95fc8f436bbbc576836655c23f (DEFAULT_TABLE table=test-table [id=815d1e3da4a748eeb827ae04f281e06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:43.460014 27575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f44b95fc8f436bbbc576836655c23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:43.460211 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.454291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.4.64.66:43509)
0810 15:24:43.454766 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.454770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.454780 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:43.454819 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.454821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.454837 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.454910 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:43.456018 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.457163 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.458469 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.459820 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.459902 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.459995 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.460077 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.460121 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":106,"threads_started":1}
I20260810 15:24:43.460250 27278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f44b95fc8f436bbbc576836655c23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:43.460471 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.454076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:43509 (local address 127.0.0.1:55414)
0810 15:24:43.454598 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.454610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.454680 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.455000 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.455003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.455005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:43.455092 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.455094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.457259 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.457260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.458378 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.458381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.459906 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.459907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.459979 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.459983 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.460154 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.460216 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":5,"thread_start_us":147,"threads_started":1}
I20260810 15:24:43.461416 27425 tablet_service.cc:1467] Processing CreateTablet for tablet 90f44b95fc8f436bbbc576836655c23f (DEFAULT_TABLE table=test-table [id=815d1e3da4a748eeb827ae04f281e06d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:24:43.461834 27425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90f44b95fc8f436bbbc576836655c23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:43.463230 27661 tablet_bootstrap.cc:492] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: Bootstrap starting.
I20260810 15:24:43.463961 27662 tablet_bootstrap.cc:492] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Bootstrap starting.
I20260810 15:24:43.464008 27661 tablet_bootstrap.cc:654] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:43.464340 27661 log.cc:824] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:43.464633 27662 tablet_bootstrap.cc:654] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:43.464998 27662 log.cc:824] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:43.465161 27661 tablet_bootstrap.cc:492] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: No bootstrap required, opened a new log
I20260810 15:24:43.465265 27661 ts_tablet_manager.cc:1397] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:24:43.465588 27665 tablet_bootstrap.cc:492] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Bootstrap starting.
I20260810 15:24:43.466271 27662 tablet_bootstrap.cc:492] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: No bootstrap required, opened a new log
I20260810 15:24:43.466297 27665 tablet_bootstrap.cc:654] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:43.466362 27662 ts_tablet_manager.cc:1397] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:24:43.466619 27665 log.cc:824] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:43.467410 27665 tablet_bootstrap.cc:492] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: No bootstrap required, opened a new log
I20260810 15:24:43.467482 27665 ts_tablet_manager.cc:1397] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:24:43.468202 27661 raft_consensus.cc:348] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.468405 27661 raft_consensus.cc:374] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:43.468446 27661 raft_consensus.cc:729] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b723fca91f334089b9103bc9b0a41022, State: Initialized, Role: FOLLOWER
I20260810 15:24:43.468591 27661 consensus_queue.cc:260] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.468878 27661 ts_tablet_manager.cc:1428] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 15:24:43.469008 27353 heartbeater.cc:503] Master 127.4.64.126:34201 was elected leader, sending a full tablet report...
I20260810 15:24:43.469175 27662 raft_consensus.cc:348] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.469378 27662 raft_consensus.cc:374] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:43.469434 27662 raft_consensus.cc:729] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01260fcddb44777a2ed327a1d1f12d1, State: Initialized, Role: FOLLOWER
I20260810 15:24:43.469568 27662 consensus_queue.cc:260] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.469878 27662 ts_tablet_manager.cc:1428] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:24:43.470031 27646 heartbeater.cc:503] Master 127.4.64.126:34201 was elected leader, sending a full tablet report...
I20260810 15:24:43.469830 27665 raft_consensus.cc:348] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.470351 27665 raft_consensus.cc:374] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:43.470400 27665 raft_consensus.cc:729] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122d863e36724f36a274eb7476329df0, State: Initialized, Role: FOLLOWER
I20260810 15:24:43.470538 27665 consensus_queue.cc:260] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.470808 27665 ts_tablet_manager.cc:1428] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:24:43.470908 27501 heartbeater.cc:503] Master 127.4.64.126:34201 was elected leader, sending a full tablet report...
W20260810 15:24:43.547154 27502 tablet.cc:2369] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:24:43.571362 27669 raft_consensus.cc:482] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:24:43.571524 27669 raft_consensus.cc:504] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.572026 27669 leader_election.cc:288] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b723fca91f334089b9103bc9b0a41022 (127.4.64.65:39521), 122d863e36724f36a274eb7476329df0 (127.4.64.66:43509)
I20260810 15:24:43.576169 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.572315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:54411 (local address 127.4.64.66:43509)
0810 15:24:43.572363 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.572366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.572526 (+ 160us) server_negotiation.cc:407] Connection header received
0810 15:24:43.572559 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.572560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.572574 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.572629 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:43.572797 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.573511 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.575167 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.575763 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.575839 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.575850 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.576035 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.576078 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:24:43.576165 27648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.572226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:39521 (local address 127.4.64.67:37647)
0810 15:24:43.572288 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.572300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.572346 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.572551 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.572552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.572554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:43.572651 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.572654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.573686 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.573687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.575060 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.575063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.575850 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.575852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.575938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.575945 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.576012 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.576064 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":58}
I20260810 15:24:43.576153 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.572338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.64.67:37647 (local address 127.4.64.65:39521)
0810 15:24:43.572384 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.572386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.572396 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:24:43.572424 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.572425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.572440 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.572523 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:24:43.572740 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.573576 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.575173 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.575759 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.575839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.575850 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.576035 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.576078 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:24:43.576179 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.572226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.66:43509 (local address 127.4.64.67:54411)
0810 15:24:43.572422 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.572431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:24:43.572484 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:24:43.572655 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:24:43.572656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:24:43.572657 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:24:43.572719 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.572721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.573600 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.573601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.575090 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:24:43.575092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.575850 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.575852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:24:43.575938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.575945 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:24:43.576012 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:24:43.576064 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":86,"thread_start_us":119,"threads_started":1}
I20260810 15:24:43.576573 27300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f44b95fc8f436bbbc576836655c23f" candidate_uuid: "f01260fcddb44777a2ed327a1d1f12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b723fca91f334089b9103bc9b0a41022" is_pre_election: true
I20260810 15:24:43.576610 27447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f44b95fc8f436bbbc576836655c23f" candidate_uuid: "f01260fcddb44777a2ed327a1d1f12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d863e36724f36a274eb7476329df0" is_pre_election: true
I20260810 15:24:43.576757 27300 raft_consensus.cc:2454] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f01260fcddb44777a2ed327a1d1f12d1 in term 0.
I20260810 15:24:43.576757 27447 raft_consensus.cc:2454] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f01260fcddb44777a2ed327a1d1f12d1 in term 0.
I20260810 15:24:43.577023 27522 leader_election.cc:302] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 122d863e36724f36a274eb7476329df0, f01260fcddb44777a2ed327a1d1f12d1; no voters:
I20260810 15:24:43.577162 27669 raft_consensus.cc:2781] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:24:43.577221 27669 raft_consensus.cc:482] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:24:43.577250 27669 raft_consensus.cc:3037] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:43.578090 27669 raft_consensus.cc:504] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.578245 27669 leader_election.cc:288] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [CANDIDATE]: Term 1 election: Requested vote from peers b723fca91f334089b9103bc9b0a41022 (127.4.64.65:39521), 122d863e36724f36a274eb7476329df0 (127.4.64.66:43509)
I20260810 15:24:43.578403 27300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f44b95fc8f436bbbc576836655c23f" candidate_uuid: "f01260fcddb44777a2ed327a1d1f12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b723fca91f334089b9103bc9b0a41022"
I20260810 15:24:43.578460 27447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90f44b95fc8f436bbbc576836655c23f" candidate_uuid: "f01260fcddb44777a2ed327a1d1f12d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "122d863e36724f36a274eb7476329df0"
I20260810 15:24:43.578518 27300 raft_consensus.cc:3037] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:43.578536 27447 raft_consensus.cc:3037] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:43.579403 27300 raft_consensus.cc:2454] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f01260fcddb44777a2ed327a1d1f12d1 in term 1.
I20260810 15:24:43.579404 27447 raft_consensus.cc:2454] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f01260fcddb44777a2ed327a1d1f12d1 in term 1.
I20260810 15:24:43.579638 27524 leader_election.cc:302] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b723fca91f334089b9103bc9b0a41022, f01260fcddb44777a2ed327a1d1f12d1; no voters:
I20260810 15:24:43.579733 27669 raft_consensus.cc:2781] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:43.579932 27669 raft_consensus.cc:686] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 1 LEADER]: Becoming Leader. State: Replica: f01260fcddb44777a2ed327a1d1f12d1, State: Running, Role: LEADER
I20260810 15:24:43.580049 27669 consensus_queue.cc:237] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } }
I20260810 15:24:43.581048 27134 catalog_manager.cc:5495] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f01260fcddb44777a2ed327a1d1f12d1 (127.4.64.67). New cstate: current_term: 1 leader_uuid: "f01260fcddb44777a2ed327a1d1f12d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f01260fcddb44777a2ed327a1d1f12d1" member_type: VOTER last_known_addr { host: "127.4.64.67" port: 35523 } health_report { overall_health: HEALTHY } } }
I20260810 15:24:43.606042 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.603405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57262 (local address 127.4.64.67:35523)
0810 15:24:43.603485 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:43.603487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:43.603496 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:24:43.603541 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:43.603542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:43.603563 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:43.603653 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:24:43.603808 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.604328 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.605130 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:43.605333 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:43.605414 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:43.605595 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:24:43.605763 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:24:43.605887 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:43.605905 (+ 18us) server_negotiation.cc:299] Negotiation successful
0810 15:24:43.605925 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 15:24:43.608174 27300 raft_consensus.cc:1261] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 1 FOLLOWER]: Refusing update from remote peer f01260fcddb44777a2ed327a1d1f12d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:43.608276 27447 raft_consensus.cc:1261] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 1 FOLLOWER]: Refusing update from remote peer f01260fcddb44777a2ed327a1d1f12d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:24:43.608608 27673 consensus_queue.cc:1035] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b723fca91f334089b9103bc9b0a41022" member_type: VOTER last_known_addr { host: "127.4.64.65" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:43.608749 27669 consensus_queue.cc:1035] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "122d863e36724f36a274eb7476329df0" member_type: VOTER last_known_addr { host: "127.4.64.66" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:24:43.611470 27678 mvcc.cc:205] Tried to move back new op lower bound from 7316993980855656448 to 7316993980744683520. Current Snapshot: MvccSnapshot[applied={T|T < 7316993980855656448}]
I20260810 15:24:43.611804 27679 mvcc.cc:205] Tried to move back new op lower bound from 7316993980855656448 to 7316993980744683520. Current Snapshot: MvccSnapshot[applied={T|T < 7316993980855656448}]
I20260810 15:24:43.612196 27680 mvcc.cc:205] Tried to move back new op lower bound from 7316993980855656448 to 7316993980744683520. Current Snapshot: MvccSnapshot[applied={T|T < 7316993980855656448}]
I20260810 15:24:43.614008 27134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:43.614117 27134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:24:43.614928 27134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e3a921e8f0a493f81088f7cce9ceb7e: Sending DeleteTablet for 3 replicas of tablet 90f44b95fc8f436bbbc576836655c23f
I20260810 15:24:43.615353 27575 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f44b95fc8f436bbbc576836655c23f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:43 UTC) from {username='slave'} at 127.0.0.1:57260
I20260810 15:24:43.615353 27278 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f44b95fc8f436bbbc576836655c23f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:43 UTC) from {username='slave'} at 127.0.0.1:40730
I20260810 15:24:43.615377 27425 tablet_service.cc:1514] Processing DeleteTablet for tablet 90f44b95fc8f436bbbc576836655c23f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:24:43 UTC) from {username='slave'} at 127.0.0.1:55414
I20260810 15:24:43.615646 27689 tablet_replica.cc:330] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022: stopping tablet replica
I20260810 15:24:43.615751 27689 raft_consensus.cc:2229] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:43.615859 27689 raft_consensus.cc:2258] T 90f44b95fc8f436bbbc576836655c23f P b723fca91f334089b9103bc9b0a41022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:43.615907 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27205
I20260810 15:24:43.616011 27688 tablet_replica.cc:330] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: stopping tablet replica
I20260810 15:24:43.616099 27688 raft_consensus.cc:2229] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:24:43.616194 27688 raft_consensus.cc:2258] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:43.616520 27687 tablet_replica.cc:330] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: stopping tablet replica
I20260810 15:24:43.616597 27687 raft_consensus.cc:2229] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:24:43.616674 27687 raft_consensus.cc:2258] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:24:43.618402 27687 ts_tablet_manager.cc:1905] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:43.620558 27687 ts_tablet_manager.cc:1918] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:43.620633 27687 log.cc:1196] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-2/wal/wals/90f44b95fc8f436bbbc576836655c23f
I20260810 15:24:43.620898 27687 ts_tablet_manager.cc:1939] T 90f44b95fc8f436bbbc576836655c23f P f01260fcddb44777a2ed327a1d1f12d1: Deleting consensus metadata
I20260810 15:24:43.621445 27120 catalog_manager.cc:4841] TS f01260fcddb44777a2ed327a1d1f12d1 (127.4.64.67:35523): tablet 90f44b95fc8f436bbbc576836655c23f (table test-table [id=815d1e3da4a748eeb827ae04f281e06d]) successfully deleted
I20260810 15:24:43.622069 27688 ts_tablet_manager.cc:1905] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:24:43.623488 27688 ts_tablet_manager.cc:1918] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:24:43.623574 27688 log.cc:1196] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Deleting WAL directory at /tmp/dist-test-task97bH11/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786375228510355-4353-0/minicluster-data/ts-1/wal/wals/90f44b95fc8f436bbbc576836655c23f
I20260810 15:24:43.623835 27688 ts_tablet_manager.cc:1939] T 90f44b95fc8f436bbbc576836655c23f P 122d863e36724f36a274eb7476329df0: Deleting consensus metadata
I20260810 15:24:43.624321 27118 catalog_manager.cc:4841] TS 122d863e36724f36a274eb7476329df0 (127.4.64.66:43509): tablet 90f44b95fc8f436bbbc576836655c23f (table test-table [id=815d1e3da4a748eeb827ae04f281e06d]) successfully deleted
W20260810 15:24:43.626566 27120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.64.65:39521 (error 108)
I20260810 15:24:43.627130 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:43.626978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.65:39521 (local address 127.0.0.1:40736)
0810 15:24:43.627043 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 15:24:43.627090 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.65:39521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 15:24:43.627211 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27356
W20260810 15:24:43.630239 27120 catalog_manager.cc:4584] TS b723fca91f334089b9103bc9b0a41022 (127.4.64.65:39521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90f44b95fc8f436bbbc576836655c23f: Network error: Client connection negotiation failed: client connection to 127.4.64.65:39521: connect: Connection refused (error 111)
I20260810 15:24:43.634634 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27504
I20260810 15:24:43.643488 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27099
2026-08-10T15:24:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (830 ms)
[----------] 6 tests from EncryptionPolicyTest (9043 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:43 dist-test-slave-9zdj krb5kdc[27693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:43 dist-test-slave-9zdj krb5kdc[27693](info): set up 2 sockets
Aug 10 15:24:43 dist-test-slave-9zdj krb5kdc[27693](info): commencing operation
krb5kdc: starting...
W20260810 15:24:44.790311 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:44 dist-test-slave-9zdj krb5kdc[27693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:44.934377 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:45.022284 27707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:45.022573 27707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:45.022615 27707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:45.023821 27707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:45.023910 27707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:45.023936 27707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:45.023965 27707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:45.023995 27707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:45.025583 27707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:45.025873 27707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:45.026075 27707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:45.031520 27717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:45.031509 27714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:45.031792 27707 server_base.cc:1034] running on GCE node
W20260810 15:24:45.031504 27715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:45.032130 27707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:45.032543 27707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:45.033713 27707 hybrid_clock.cc:648] HybridClock initialized: now 1786375485033665 us; error 55 us; skew 500 ppm
Aug 10 15:24:45 dist-test-slave-9zdj krb5kdc[27693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:45.042410 27707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:45.043051 27707 webserver.cc:460] Webserver started at http://127.0.0.1:38751/ using document root <none> and password file <none>
I20260810 15:24:45.043308 27707 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:45.043363 27707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:45.043478 27707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:45.044706 27707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242"
format_stamp: "Formatted at 2026-08-10 15:24:45 on dist-test-slave-9zdj"
server_key: "073d47446b8fef58c48006c272104042"
server_key_iv: "24f5a2061dc465b4987258ce73f38301"
server_key_version: "encryptionkey@0"
I20260810 15:24:45.045055 27707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242"
format_stamp: "Formatted at 2026-08-10 15:24:45 on dist-test-slave-9zdj"
server_key: "073d47446b8fef58c48006c272104042"
server_key_iv: "24f5a2061dc465b4987258ce73f38301"
server_key_version: "encryptionkey@0"
I20260810 15:24:45.046954 27707 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 15:24:45.047895 27724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:45.048142 27707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:24:45.048259 27707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242"
format_stamp: "Formatted at 2026-08-10 15:24:45 on dist-test-slave-9zdj"
server_key: "073d47446b8fef58c48006c272104042"
server_key_iv: "24f5a2061dc465b4987258ce73f38301"
server_key_version: "encryptionkey@0"
I20260810 15:24:45.048327 27707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:45.066645 27707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:45.069283 27707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:45.069415 27707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:45.074107 27707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42981
I20260810 15:24:45.074179 27787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42981 every 8 connection(s)
I20260810 15:24:45.074558 27707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:45.075812 27788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:45.078485 27788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242: Bootstrap starting.
I20260810 15:24:45.079185 27788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:45.079510 27788 log.cc:824] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:45.080451 27788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242: No bootstrap required, opened a new log
I20260810 15:24:45.081350 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27707
I20260810 15:24:45.081481 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:45.081738 4353 external_mini_cluster.cc:1428] Setting key 2d176d6e41a5c572eeaa2ce8583a6a68
I20260810 15:24:45.082840 27788 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 15:24:45.083024 27788 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:45.083070 27788 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e1dcbefd5a4ec8a3d4fcb35247e242, State: Initialized, Role: FOLLOWER
I20260810 15:24:45.083206 27788 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 15:24:45.083300 27788 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:45.083352 27788 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:45.083429 27788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:45.084395 27788 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 15:24:45.084532 27788 leader_election.cc:302] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2e1dcbefd5a4ec8a3d4fcb35247e242; no voters:
I20260810 15:24:45.084750 27788 leader_election.cc:288] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:45.084777 27793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:45.084908 27793 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [term 1 LEADER]: Becoming Leader. State: Replica: e2e1dcbefd5a4ec8a3d4fcb35247e242, State: Running, Role: LEADER
I20260810 15:24:45.085001 27793 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 15:24:45.085060 27788 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:45.085397 27793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2e1dcbefd5a4ec8a3d4fcb35247e242. Latest consensus state: current_term: 1 leader_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } } }
I20260810 15:24:45.085470 27793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:45.086099 27801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:45.086488 27794 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2e1dcbefd5a4ec8a3d4fcb35247e242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } } }
I20260810 15:24:45.086638 27794 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:45.086656 27801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:45.088549 27801 catalog_manager.cc:1347] Generated new cluster ID: 80752340fdf44ac99a434aef89bde5ab
I20260810 15:24:45.088624 27801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:24:45 dist-test-slave-9zdj krb5kdc[27693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:24:45.094652 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:45.082427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33777 (local address 127.0.0.1:42981)
0810 15:24:45.082656 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:45.082665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:45.082698 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:24:45.082807 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:45.082812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:45.082860 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:45.083016 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:45.083950 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:45.085115 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:45.086356 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:45.087015 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:45.087162 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:45.090222 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:45.090227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:45.090227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:45.090242 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:45.092413 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:45.092958 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:45.092963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:45.092963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:45.093121 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:45.093393 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:45.093394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:45.093394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:45.093802 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:45.094012 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:45.094106 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 15:24:45.094227 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":113,"threads_started":1}
I20260810 15:24:45.096210 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:45.106129 27792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:45.097824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49549 (local address 127.0.0.1:42981)
0810 15:24:45.097875 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:45.097877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:45.098011 (+ 134us) server_negotiation.cc:407] Connection header received
0810 15:24:45.098110 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:45.098111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:45.098124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:45.098155 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:45.098915 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:45.099734 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:45.100838 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:45.101172 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:45.101220 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:45.102925 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:45.102926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:45.102927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:45.102934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:45.104607 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:45.105081 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:45.105082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:45.105082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:45.105162 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:45.105410 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:45.105411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:45.105411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:45.105553 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:45.105668 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:45.105946 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 15:24:45.106020 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:24:45.109428 27801 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:45.110169 27801 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:45.119980 27801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2e1dcbefd5a4ec8a3d4fcb35247e242: Generated new TSK 0
I20260810 15:24:45.120235 27801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:45.125758 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27707
W20260810 15:24:45.133204 27691 connection.cc:535] client connection to 127.0.0.1:42981 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42981): Connection reset by peer (error 104)
2026-08-10T15:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:45 dist-test-slave-9zdj krb5kdc[27826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:45 dist-test-slave-9zdj krb5kdc[27826](info): set up 2 sockets
Aug 10 15:24:45 dist-test-slave-9zdj krb5kdc[27826](info): commencing operation
krb5kdc: starting...
W20260810 15:24:46.319492 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:46 dist-test-slave-9zdj krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:46.470026 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:46.558451 27840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:46.558699 27840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:46.558727 27840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:46.560014 27840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:46.560077 27840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:46.560092 27840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:46.560105 27840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:46.560153 27840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:46.561915 27840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:46.562237 27840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:46.562502 27840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:46.568238 27846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:46.568243 27850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:46.568282 27847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:46.568665 27840 server_base.cc:1034] running on GCE node
I20260810 15:24:46.568984 27840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:46.569275 27840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:46.570433 27840 hybrid_clock.cc:648] HybridClock initialized: now 1786375486570411 us; error 39 us; skew 500 ppm
Aug 10 15:24:46 dist-test-slave-9zdj krb5kdc[27826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:46.573225 27840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:46.573743 27840 webserver.cc:460] Webserver started at http://127.0.0.1:35361/ using document root <none> and password file <none>
I20260810 15:24:46.573956 27840 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:46.573999 27840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:46.574097 27840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:46.575176 27840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e788373bea9d4b03a659dadd87919740"
format_stamp: "Formatted at 2026-08-10 15:24:46 on dist-test-slave-9zdj"
server_key: "d4ca40f94a3039307f2bb554e5717683"
server_key_iv: "89de1f17579eeae6b9cad7b0a75ac849"
server_key_version: "encryptionkey@0"
I20260810 15:24:46.575498 27840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e788373bea9d4b03a659dadd87919740"
format_stamp: "Formatted at 2026-08-10 15:24:46 on dist-test-slave-9zdj"
server_key: "d4ca40f94a3039307f2bb554e5717683"
server_key_iv: "89de1f17579eeae6b9cad7b0a75ac849"
server_key_version: "encryptionkey@0"
I20260810 15:24:46.577102 27840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:24:46.578069 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:46.578388 27840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:46.578478 27840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e788373bea9d4b03a659dadd87919740"
format_stamp: "Formatted at 2026-08-10 15:24:46 on dist-test-slave-9zdj"
server_key: "d4ca40f94a3039307f2bb554e5717683"
server_key_iv: "89de1f17579eeae6b9cad7b0a75ac849"
server_key_version: "encryptionkey@0"
I20260810 15:24:46.578581 27840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:46.590929 27840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:46.593199 27840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:46.593400 27840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:46.597523 27840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36617
I20260810 15:24:46.597596 27916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36617 every 8 connection(s)
I20260810 15:24:46.598067 27840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:46.599174 27917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:46.601539 27917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740: Bootstrap starting.
I20260810 15:24:46.602234 27917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:46.602489 27917 log.cc:824] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:46.603238 27917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740: No bootstrap required, opened a new log
I20260810 15:24:46.604897 27917 raft_consensus.cc:348] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e788373bea9d4b03a659dadd87919740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36617 } }
I20260810 15:24:46.605049 27917 raft_consensus.cc:374] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:46.605072 27917 raft_consensus.cc:729] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e788373bea9d4b03a659dadd87919740, State: Initialized, Role: FOLLOWER
I20260810 15:24:46.605198 27917 consensus_queue.cc:260] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e788373bea9d4b03a659dadd87919740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36617 } }
I20260810 15:24:46.605300 27917 raft_consensus.cc:388] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:46.605340 27917 raft_consensus.cc:482] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:46.605392 27917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:46.606071 27917 raft_consensus.cc:504] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e788373bea9d4b03a659dadd87919740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36617 } }
I20260810 15:24:46.606205 27917 leader_election.cc:302] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e788373bea9d4b03a659dadd87919740; no voters:
I20260810 15:24:46.606405 27917 leader_election.cc:288] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:46.606465 27920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:46.606726 27920 raft_consensus.cc:686] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [term 1 LEADER]: Becoming Leader. State: Replica: e788373bea9d4b03a659dadd87919740, State: Running, Role: LEADER
I20260810 15:24:46.606798 27917 sys_catalog.cc:564] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:46.606865 27920 consensus_queue.cc:237] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e788373bea9d4b03a659dadd87919740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36617 } }
I20260810 15:24:46.607223 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27840
I20260810 15:24:46.607373 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:46.607530 27921 sys_catalog.cc:455] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e788373bea9d4b03a659dadd87919740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e788373bea9d4b03a659dadd87919740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36617 } } }
I20260810 15:24:46.607633 27921 sys_catalog.cc:458] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:46.607640 4353 external_mini_cluster.cc:1428] Setting key fee06ad3601a131a55019f7ecf5b5ca9
I20260810 15:24:46.608026 27922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e788373bea9d4b03a659dadd87919740. Latest consensus state: current_term: 1 leader_uuid: "e788373bea9d4b03a659dadd87919740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e788373bea9d4b03a659dadd87919740" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36617 } } }
I20260810 15:24:46.608503 27929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:46.608527 27922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:46.609238 27929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:46.611122 27929 catalog_manager.cc:1347] Generated new cluster ID: 15027a1dd8b44e11ac4d9fa3a40efd2d
I20260810 15:24:46.611195 27929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:46.614437 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:46.608146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59111 (local address 127.0.0.1:36617)
0810 15:24:46.608451 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:46.608462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:46.608627 (+ 165us) server_negotiation.cc:407] Connection header received
0810 15:24:46.608853 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:46.608862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:46.608937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:46.609093 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:46.609323 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:46.610578 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:46.611876 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:46.612536 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:46.612670 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:46.613581 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:46.613585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:46.613585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:46.613865 (+ 280us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:46.613907 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:46.613924 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:46.614007 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:46.614166 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 15:24:46.614199 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":88,"threads_started":1}
I20260810 15:24:46.615568 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:46.621181 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:46.617341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57421 (local address 127.0.0.1:36617)
0810 15:24:46.617394 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:46.617396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:46.617508 (+ 112us) server_negotiation.cc:407] Connection header received
0810 15:24:46.617578 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:46.617579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:46.617649 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:46.617707 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:46.617907 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:46.618659 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:46.619697 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:46.620029 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:46.620070 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:46.620901 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:46.620903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:46.620903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:46.620909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:46.620923 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:46.620928 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:46.620960 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:46.621077 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 15:24:46.621092 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:24:46.639503 27929 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:46.640219 27929 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:46.654105 27929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e788373bea9d4b03a659dadd87919740: Generated new TSK 0
I20260810 15:24:46.654372 27929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:46.657604 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27840
W20260810 15:24:46.666375 27824 connection.cc:535] client connection to 127.0.0.1:36617 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36617): Connection reset by peer (error 104)
2026-08-10T15:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:46 dist-test-slave-9zdj krb5kdc[27953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:46 dist-test-slave-9zdj krb5kdc[27953](info): set up 2 sockets
Aug 10 15:24:46 dist-test-slave-9zdj krb5kdc[27953](info): commencing operation
krb5kdc: starting...
W20260810 15:24:47.873903 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:47 dist-test-slave-9zdj krb5kdc[27953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:48.009858 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:48.097218 27969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:48.097451 27969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:48.097476 27969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:48.098661 27969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:48.098707 27969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:48.098722 27969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:48.098737 27969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:48.098749 27969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:48.100226 27969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:48.100458 27969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:48.100669 27969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:48.106140 27976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:48.106127 27978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:48.106140 27975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:48.106490 27969 server_base.cc:1034] running on GCE node
I20260810 15:24:48.106756 27969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:48.107044 27969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:48.108204 27969 hybrid_clock.cc:648] HybridClock initialized: now 1786375488108181 us; error 49 us; skew 500 ppm
Aug 10 15:24:48 dist-test-slave-9zdj krb5kdc[27953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:48.111192 27969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:48.111742 27969 webserver.cc:460] Webserver started at http://127.0.0.1:37635/ using document root <none> and password file <none>
I20260810 15:24:48.112015 27969 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:48.112067 27969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:48.112207 27969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:48.113440 27969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e2ab8693ed7c4cdb9ccd781b91615e53"
format_stamp: "Formatted at 2026-08-10 15:24:48 on dist-test-slave-9zdj"
server_key: "43f5314b1fcea8c1363066334c09cbf4"
server_key_iv: "c6f6bc7cea609fcbc544fbca7ec9a336"
server_key_version: "encryptionkey@0"
I20260810 15:24:48.113921 27969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e2ab8693ed7c4cdb9ccd781b91615e53"
format_stamp: "Formatted at 2026-08-10 15:24:48 on dist-test-slave-9zdj"
server_key: "43f5314b1fcea8c1363066334c09cbf4"
server_key_iv: "c6f6bc7cea609fcbc544fbca7ec9a336"
server_key_version: "encryptionkey@0"
I20260810 15:24:48.115507 27969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:48.116595 27985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:48.116858 27969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:48.116986 27969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e2ab8693ed7c4cdb9ccd781b91615e53"
format_stamp: "Formatted at 2026-08-10 15:24:48 on dist-test-slave-9zdj"
server_key: "43f5314b1fcea8c1363066334c09cbf4"
server_key_iv: "c6f6bc7cea609fcbc544fbca7ec9a336"
server_key_version: "encryptionkey@0"
I20260810 15:24:48.117074 27969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:48.120821 27969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:48.123334 27969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:48.123519 27969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:48.128059 27969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36149
I20260810 15:24:48.128106 28043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36149 every 8 connection(s)
I20260810 15:24:48.128510 27969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:48.129858 28044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:48.132416 28044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53: Bootstrap starting.
I20260810 15:24:48.133050 28044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:48.133327 28044 log.cc:824] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:48.134168 28044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53: No bootstrap required, opened a new log
I20260810 15:24:48.135867 28044 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260810 15:24:48.136035 28044 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:48.136060 28044 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ab8693ed7c4cdb9ccd781b91615e53, State: Initialized, Role: FOLLOWER
I20260810 15:24:48.136231 28044 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260810 15:24:48.136329 28044 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:48.136353 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 27969
I20260810 15:24:48.136376 28044 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:48.136436 28044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:48.136482 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:48.136705 4353 external_mini_cluster.cc:1428] Setting key 69df1b6135e482eb1c1a4c196623e1de
I20260810 15:24:48.137204 28044 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260810 15:24:48.137354 28044 leader_election.cc:302] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ab8693ed7c4cdb9ccd781b91615e53; no voters:
I20260810 15:24:48.137548 28044 leader_election.cc:288] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:48.137696 28049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:48.137887 28044 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:48.137887 28049 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [term 1 LEADER]: Becoming Leader. State: Replica: e2ab8693ed7c4cdb9ccd781b91615e53, State: Running, Role: LEADER
I20260810 15:24:48.138513 28049 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260810 15:24:48.139268 28050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } } }
I20260810 15:24:48.139387 28050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:48.139339 28051 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ab8693ed7c4cdb9ccd781b91615e53. Latest consensus state: current_term: 1 leader_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab8693ed7c4cdb9ccd781b91615e53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } } }
I20260810 15:24:48.139439 28051 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:48.140069 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:48.137355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48493 (local address 127.0.0.1:36149)
0810 15:24:48.137626 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:48.137674 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:48.137717 (+ 43us) server_negotiation.cc:407] Connection header received
0810 15:24:48.137855 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:48.137864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:48.137902 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:48.138065 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:48.138990 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:48.138995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:48.138995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:48.139000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:48.139056 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:48.139075 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:48.139420 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:48.139462 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:24:48.139514 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":139,"threads_started":1}
W20260810 15:24:48.140645 28065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:24:48.140718 28065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:24:48.140782 28060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:48.140952 28060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:48.141307 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:48.142829 28060 catalog_manager.cc:1347] Generated new cluster ID: 056cb7dc15b84afea2f87adebbb15b52
I20260810 15:24:48.142900 28060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:48.144498 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:48.143060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35081 (local address 127.0.0.1:36149)
0810 15:24:48.143103 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:48.143106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:48.143428 (+ 322us) server_negotiation.cc:407] Connection header received
0810 15:24:48.143517 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:48.143518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:48.143523 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:48.143553 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:48.144272 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:48.144273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:48.144273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:48.144277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:48.144290 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:48.144295 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:48.144321 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:48.144425 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 15:24:48.144438 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:24:48.154546 28060 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:48.154770 28060 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:48.164881 28060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2ab8693ed7c4cdb9ccd781b91615e53: Generated new TSK 0
I20260810 15:24:48.165184 28060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:48.167336 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 27969
2026-08-10T15:24:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:48 dist-test-slave-9zdj krb5kdc[28076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:48 dist-test-slave-9zdj krb5kdc[28076](info): set up 2 sockets
Aug 10 15:24:48 dist-test-slave-9zdj krb5kdc[28076](info): commencing operation
krb5kdc: starting...
W20260810 15:24:49.312831 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:49 dist-test-slave-9zdj krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:49.457489 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:49.548908 28091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:49.549209 28091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:49.549254 28091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:49.550655 28091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:49.550745 28091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:49.550770 28091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:49.550794 28091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:49.550828 28091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:49.552637 28091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:49.552997 28091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:49.553294 28091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:49.559646 28100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:49.559648 28101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:49.559928 28091 server_base.cc:1034] running on GCE node
W20260810 15:24:49.559861 28104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:49.560359 28091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:49.560705 28091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:49.561880 28091 hybrid_clock.cc:648] HybridClock initialized: now 1786375489561862 us; error 43 us; skew 500 ppm
Aug 10 15:24:49 dist-test-slave-9zdj krb5kdc[28076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:49.564643 28091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:49.565162 28091 webserver.cc:460] Webserver started at http://127.0.0.1:37623/ using document root <none> and password file <none>
I20260810 15:24:49.565431 28091 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:49.565506 28091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:49.565709 28091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:49.566865 28091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "d1c3b34a65574259881e4bca6a885108"
format_stamp: "Formatted at 2026-08-10 15:24:49 on dist-test-slave-9zdj"
server_key: "ccfa7a16418086c6689d2caf6b886cba"
server_key_iv: "ef77653afb336f2c5cb0b92773a6c03e"
server_key_version: "encryptionkey@0"
I20260810 15:24:49.567250 28091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "d1c3b34a65574259881e4bca6a885108"
format_stamp: "Formatted at 2026-08-10 15:24:49 on dist-test-slave-9zdj"
server_key: "ccfa7a16418086c6689d2caf6b886cba"
server_key_iv: "ef77653afb336f2c5cb0b92773a6c03e"
server_key_version: "encryptionkey@0"
I20260810 15:24:49.568702 28091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:24:49.569701 28113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:49.569898 28091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:49.570019 28091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "d1c3b34a65574259881e4bca6a885108"
format_stamp: "Formatted at 2026-08-10 15:24:49 on dist-test-slave-9zdj"
server_key: "ccfa7a16418086c6689d2caf6b886cba"
server_key_iv: "ef77653afb336f2c5cb0b92773a6c03e"
server_key_version: "encryptionkey@0"
I20260810 15:24:49.570108 28091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:49.587929 28091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:49.590466 28091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:49.590662 28091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:49.595171 28091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45167
I20260810 15:24:49.595273 28170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45167 every 8 connection(s)
I20260810 15:24:49.595643 28091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:49.596743 28171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:49.599327 28171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108: Bootstrap starting.
I20260810 15:24:49.599964 28171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:49.600234 28171 log.cc:824] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:49.601120 28171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108: No bootstrap required, opened a new log
I20260810 15:24:49.602924 28171 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b34a65574259881e4bca6a885108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260810 15:24:49.603071 28171 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:49.603094 28171 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c3b34a65574259881e4bca6a885108, State: Initialized, Role: FOLLOWER
I20260810 15:24:49.603240 28171 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b34a65574259881e4bca6a885108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260810 15:24:49.603323 28171 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:49.603346 28171 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:49.603379 28171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:49.603920 28171 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b34a65574259881e4bca6a885108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260810 15:24:49.604014 28171 leader_election.cc:302] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c3b34a65574259881e4bca6a885108; no voters:
I20260810 15:24:49.604180 28171 leader_election.cc:288] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:49.604236 28174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:49.604355 28174 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [term 1 LEADER]: Becoming Leader. State: Replica: d1c3b34a65574259881e4bca6a885108, State: Running, Role: LEADER
I20260810 15:24:49.604432 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28091
I20260810 15:24:49.604462 28171 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:49.604501 28174 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b34a65574259881e4bca6a885108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } }
I20260810 15:24:49.604547 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:49.604780 4353 external_mini_cluster.cc:1428] Setting key e6d0503c6baaacec42b7068541a24690
I20260810 15:24:49.604792 28174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c3b34a65574259881e4bca6a885108. Latest consensus state: current_term: 1 leader_uuid: "d1c3b34a65574259881e4bca6a885108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b34a65574259881e4bca6a885108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } } }
I20260810 15:24:49.604856 28174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:49.604983 28175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c3b34a65574259881e4bca6a885108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c3b34a65574259881e4bca6a885108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45167 } } }
I20260810 15:24:49.605067 28175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:49.605085 28181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:49.605717 28181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:49.607475 28181 catalog_manager.cc:1347] Generated new cluster ID: 9009808abeb341388d32a177ae9c2c77
I20260810 15:24:49.607534 28181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:24:49 dist-test-slave-9zdj krb5kdc[28076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:24:49.616483 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:49.605518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48129 (local address 127.0.0.1:45167)
0810 15:24:49.606321 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:49.606329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:49.606355 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:24:49.606489 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:49.606496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:49.606553 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:49.606693 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:49.607641 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:49.608595 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:49.609394 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:49.609952 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:49.610080 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:49.612560 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:49.612564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:49.612565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:49.612578 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:49.614582 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:49.614982 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:49.614986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:49.614987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:49.615173 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:49.615457 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:49.615458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:49.615458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:49.615737 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:49.615917 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:49.616052 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:24:49.616227 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":119,"threads_started":1}
I20260810 15:24:49.617642 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:49.626358 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:49.619077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56339 (local address 127.0.0.1:45167)
0810 15:24:49.619119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:49.619121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:49.619275 (+ 154us) server_negotiation.cc:407] Connection header received
0810 15:24:49.619335 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:49.619394 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:49.619405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:49.619575 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:49.620217 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:49.620961 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:49.622034 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:49.622274 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:49.622355 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:49.623660 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:49.623662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:49.623662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:49.623671 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:49.625145 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:49.625508 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:49.625509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:49.625510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:49.625571 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:49.625818 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:49.625819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:49.625819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:49.625937 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:49.626060 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:49.626158 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 15:24:49.626246 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:24:49.650486 28181 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:49.651239 28181 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:49.658651 28181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1c3b34a65574259881e4bca6a885108: Generated new TSK 0
I20260810 15:24:49.658924 28181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:49.672358 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28091
2026-08-10T15:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:49 dist-test-slave-9zdj krb5kdc[28207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:49 dist-test-slave-9zdj krb5kdc[28207](info): set up 2 sockets
Aug 10 15:24:49 dist-test-slave-9zdj krb5kdc[28207](info): commencing operation
krb5kdc: starting...
W20260810 15:24:50.814112 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:50 dist-test-slave-9zdj krb5kdc[28207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:50.953464 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:51.038476 28220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:51.038714 28220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:51.038739 28220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:51.039951 28220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:51.040004 28220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:51.040019 28220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:51.040032 28220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:51.040084 28220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:51.041801 28220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:51.042081 28220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:51.042342 28220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:51.047916 28226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:51.048039 28220 server_base.cc:1034] running on GCE node
W20260810 15:24:51.047916 28225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:51.047914 28228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:51.048521 28220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:51.048903 28220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:51.050124 28220 hybrid_clock.cc:648] HybridClock initialized: now 1786375491050101 us; error 45 us; skew 500 ppm
Aug 10 15:24:51 dist-test-slave-9zdj krb5kdc[28207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:51.064826 28220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:51.065605 28220 webserver.cc:460] Webserver started at http://127.0.0.1:43165/ using document root <none> and password file <none>
I20260810 15:24:51.065970 28220 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:51.066026 28220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:51.066205 28220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:51.067437 28220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "54b8fd5de49441d98e40d9b3240a5fee"
format_stamp: "Formatted at 2026-08-10 15:24:51 on dist-test-slave-9zdj"
server_key: "36da6cf1b4d83561ab7234d4c5ab0003"
server_key_iv: "bda5c7138b989b547b682032ea50e5d2"
server_key_version: "encryptionkey@0"
I20260810 15:24:51.067847 28220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "54b8fd5de49441d98e40d9b3240a5fee"
format_stamp: "Formatted at 2026-08-10 15:24:51 on dist-test-slave-9zdj"
server_key: "36da6cf1b4d83561ab7234d4c5ab0003"
server_key_iv: "bda5c7138b989b547b682032ea50e5d2"
server_key_version: "encryptionkey@0"
I20260810 15:24:51.069437 28220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:24:51.070346 28236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:51.070554 28220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:51.070669 28220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "54b8fd5de49441d98e40d9b3240a5fee"
format_stamp: "Formatted at 2026-08-10 15:24:51 on dist-test-slave-9zdj"
server_key: "36da6cf1b4d83561ab7234d4c5ab0003"
server_key_iv: "bda5c7138b989b547b682032ea50e5d2"
server_key_version: "encryptionkey@0"
I20260810 15:24:51.070763 28220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:51.088667 28220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:51.091451 28220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:51.091640 28220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:51.096204 28220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45893
I20260810 15:24:51.096248 28294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45893 every 8 connection(s)
I20260810 15:24:51.096671 28220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:51.097801 28295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:51.099978 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28220
I20260810 15:24:51.100243 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:51.100353 28295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee: Bootstrap starting.
I20260810 15:24:51.100519 4353 external_mini_cluster.cc:1428] Setting key 1cf046db9ef21f4b81581efeef812a29
I20260810 15:24:51.101142 28295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:51.101501 28295 log.cc:824] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:51.102341 28295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee: No bootstrap required, opened a new log
I20260810 15:24:51.104473 28295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b8fd5de49441d98e40d9b3240a5fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260810 15:24:51.104684 28295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:51.104736 28295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b8fd5de49441d98e40d9b3240a5fee, State: Initialized, Role: FOLLOWER
I20260810 15:24:51.104883 28295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b8fd5de49441d98e40d9b3240a5fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260810 15:24:51.104969 28295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:51.104995 28295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:51.105072 28295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:51.105949 28295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b8fd5de49441d98e40d9b3240a5fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260810 15:24:51.106083 28295 leader_election.cc:302] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54b8fd5de49441d98e40d9b3240a5fee; no voters:
I20260810 15:24:51.106294 28295 leader_election.cc:288] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:51.106353 28301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:51.106537 28301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [term 1 LEADER]: Becoming Leader. State: Replica: 54b8fd5de49441d98e40d9b3240a5fee, State: Running, Role: LEADER
I20260810 15:24:51.106617 28295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:51.106760 28301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b8fd5de49441d98e40d9b3240a5fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260810 15:24:51.107213 28301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54b8fd5de49441d98e40d9b3240a5fee. Latest consensus state: current_term: 1 leader_uuid: "54b8fd5de49441d98e40d9b3240a5fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b8fd5de49441d98e40d9b3240a5fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } } }
I20260810 15:24:51.107409 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:51.101157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54217 (local address 127.0.0.1:45893)
0810 15:24:51.101396 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:51.101404 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:51.101433 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:24:51.102014 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:51.102022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:51.102078 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:51.102222 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:51.102567 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:51.103534 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:51.104751 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:51.105338 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:51.105535 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:51.106338 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:51.106372 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:51.106372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:51.106635 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:51.106682 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:51.106699 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:51.107055 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:51.107091 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 15:24:51.107173 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260810 15:24:51.107366 28303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54b8fd5de49441d98e40d9b3240a5fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b8fd5de49441d98e40d9b3240a5fee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } } }
I20260810 15:24:51.107432 28301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:51.107692 28303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:51.108031 28313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:51.108429 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:51.108695 28313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:51.110409 28313 catalog_manager.cc:1347] Generated new cluster ID: 36e58ff1c47f40a2ac7306ff62e50cf7
I20260810 15:24:51.110466 28313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:51.117976 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:51.114070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36021 (local address 127.0.0.1:45893)
0810 15:24:51.114121 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:51.114123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:51.114279 (+ 156us) server_negotiation.cc:407] Connection header received
0810 15:24:51.114316 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:51.114317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:51.114332 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:51.114418 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:51.114590 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:51.115372 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:51.116435 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:51.116791 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:51.116884 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:51.117694 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:51.117696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:51.117696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:51.117702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:51.117720 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:51.117725 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:51.117787 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:51.117845 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 15:24:51.117877 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:24:51.123358 28313 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:51.123951 28313 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:51.128912 28313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54b8fd5de49441d98e40d9b3240a5fee: Generated new TSK 0
I20260810 15:24:51.129133 28313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:51.135550 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28220
2026-08-10T15:24:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:51 dist-test-slave-9zdj krb5kdc[28330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:51 dist-test-slave-9zdj krb5kdc[28330](info): set up 2 sockets
Aug 10 15:24:51 dist-test-slave-9zdj krb5kdc[28330](info): commencing operation
krb5kdc: starting...
W20260810 15:24:52.261262 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:52 dist-test-slave-9zdj krb5kdc[28330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:52.401237 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:52.487190 28344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:52.487409 28344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:52.487432 28344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:52.488626 28344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:52.488672 28344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:52.488685 28344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:52.488698 28344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:52.488762 28344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:52.490248 28344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:52.490527 28344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:52.490783 28344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:52.496060 28350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:52.496073 28351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:52.496193 28344 server_base.cc:1034] running on GCE node
W20260810 15:24:52.496073 28353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:52.496709 28344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:52.496969 28344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:52.498135 28344 hybrid_clock.cc:648] HybridClock initialized: now 1786375492498113 us; error 32 us; skew 500 ppm
Aug 10 15:24:52 dist-test-slave-9zdj krb5kdc[28330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:52.500744 28344 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:52.501266 28344 webserver.cc:460] Webserver started at http://127.0.0.1:40217/ using document root <none> and password file <none>
I20260810 15:24:52.501504 28344 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:52.501550 28344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:52.501725 28344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:52.502815 28344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "b8d29a860ea34f5e85827335e5a030c7"
format_stamp: "Formatted at 2026-08-10 15:24:52 on dist-test-slave-9zdj"
server_key: "030c18b4d11f1d0c5b6ec59cfb325575"
server_key_iv: "d4633e3199d9dcf89726cf1f2fc849b5"
server_key_version: "encryptionkey@0"
I20260810 15:24:52.503191 28344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "b8d29a860ea34f5e85827335e5a030c7"
format_stamp: "Formatted at 2026-08-10 15:24:52 on dist-test-slave-9zdj"
server_key: "030c18b4d11f1d0c5b6ec59cfb325575"
server_key_iv: "d4633e3199d9dcf89726cf1f2fc849b5"
server_key_version: "encryptionkey@0"
I20260810 15:24:52.504608 28344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:24:52.505395 28360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:52.505600 28344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:52.505726 28344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "b8d29a860ea34f5e85827335e5a030c7"
format_stamp: "Formatted at 2026-08-10 15:24:52 on dist-test-slave-9zdj"
server_key: "030c18b4d11f1d0c5b6ec59cfb325575"
server_key_iv: "d4633e3199d9dcf89726cf1f2fc849b5"
server_key_version: "encryptionkey@0"
I20260810 15:24:52.505798 28344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:52.509110 28344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:52.511482 28344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:52.511598 28344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:52.515480 28344 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38703
I20260810 15:24:52.515532 28416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38703 every 8 connection(s)
I20260810 15:24:52.515885 28344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:52.516916 28417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:52.517484 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28344
I20260810 15:24:52.517601 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:52.517901 4353 external_mini_cluster.cc:1428] Setting key 2926329efb3537267144efb6d1187f5f
I20260810 15:24:52.519855 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7: Bootstrap starting.
I20260810 15:24:52.520408 28417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:52.520649 28417 log.cc:824] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:52.521493 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7: No bootstrap required, opened a new log
I20260810 15:24:52.522883 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:52.518492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34339 (local address 127.0.0.1:38703)
0810 15:24:52.520900 (+ 2408us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:52.520910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:52.520948 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:24:52.521460 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:52.521468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:52.521505 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:52.521700 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:52.522398 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:52.522401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:52.522401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:52.522405 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:52.522443 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:52.522453 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:52.522524 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:52.522607 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 15:24:52.522666 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":68,"threads_started":1}
I20260810 15:24:52.523923 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:52.524178 28417 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d29a860ea34f5e85827335e5a030c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260810 15:24:52.524346 28417 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:52.524385 28417 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d29a860ea34f5e85827335e5a030c7, State: Initialized, Role: FOLLOWER
I20260810 15:24:52.524518 28417 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d29a860ea34f5e85827335e5a030c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260810 15:24:52.524605 28417 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:52.524648 28417 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:52.524711 28417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:52.525545 28417 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d29a860ea34f5e85827335e5a030c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260810 15:24:52.525682 28417 leader_election.cc:302] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8d29a860ea34f5e85827335e5a030c7; no voters:
I20260810 15:24:52.525867 28417 leader_election.cc:288] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:52.525892 28429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:52.526091 28429 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [term 1 LEADER]: Becoming Leader. State: Replica: b8d29a860ea34f5e85827335e5a030c7, State: Running, Role: LEADER
I20260810 15:24:52.526216 28417 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:52.526234 28429 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d29a860ea34f5e85827335e5a030c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260810 15:24:52.526579 28429 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8d29a860ea34f5e85827335e5a030c7. Latest consensus state: current_term: 1 leader_uuid: "b8d29a860ea34f5e85827335e5a030c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d29a860ea34f5e85827335e5a030c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } } }
I20260810 15:24:52.526763 28429 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:52.526863 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:52.525576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37041 (local address 127.0.0.1:38703)
0810 15:24:52.525653 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:52.525655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:52.525731 (+ 76us) server_negotiation.cc:407] Connection header received
0810 15:24:52.525805 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:52.525806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:52.525812 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:52.525868 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:52.526524 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:52.526525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:52.526525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:52.526530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:52.526545 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:52.526550 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:52.526613 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:52.526739 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 15:24:52.526754 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 15:24:52.526590 28431 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8d29a860ea34f5e85827335e5a030c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8d29a860ea34f5e85827335e5a030c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } } }
I20260810 15:24:52.527014 28431 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:52.527412 28439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:52.527961 28439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:52.529119 28449 mvcc.cc:205] Tried to move back new op lower bound from 7316994017397964800 to 7316994017387790336. Current Snapshot: MvccSnapshot[applied={T|T < 7316994017397964800}]
I20260810 15:24:52.529603 28439 catalog_manager.cc:1347] Generated new cluster ID: 9ae73598e5914087897a0c5d63b75a1f
I20260810 15:24:52.529688 28439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:52.546142 28439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:52.546375 28439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:52.555943 28439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8d29a860ea34f5e85827335e5a030c7: Generated new TSK 0
I20260810 15:24:52.556196 28439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:52.561720 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28344
2026-08-10T15:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:52 dist-test-slave-9zdj krb5kdc[28454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:52 dist-test-slave-9zdj krb5kdc[28454](info): set up 2 sockets
Aug 10 15:24:52 dist-test-slave-9zdj krb5kdc[28454](info): commencing operation
krb5kdc: starting...
W20260810 15:24:53.687639 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:53 dist-test-slave-9zdj krb5kdc[28454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:53.834170 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:53.921294 28468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:53.921515 28468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:53.921541 28468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:53.922758 28468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:53.922811 28468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:53.922825 28468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:53.922839 28468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:53.922851 28468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:53.924286 28468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:53.924495 28468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:53.924679 28468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:53.930204 28475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:53.930197 28478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:53.930579 28468 server_base.cc:1034] running on GCE node
W20260810 15:24:53.930197 28474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:53.930981 28468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:53.931370 28468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:53.932574 28468 hybrid_clock.cc:648] HybridClock initialized: now 1786375493932542 us; error 48 us; skew 500 ppm
Aug 10 15:24:53 dist-test-slave-9zdj krb5kdc[28454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:53.937054 28468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:53.937739 28468 webserver.cc:460] Webserver started at http://127.0.0.1:33457/ using document root <none> and password file <none>
I20260810 15:24:53.938040 28468 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:53.938125 28468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:53.938292 28468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:53.939896 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "113cba18140d4843b0df81e4ff428a5c"
format_stamp: "Formatted at 2026-08-10 15:24:53 on dist-test-slave-9zdj"
server_key: "881701b1613b332ada5751193aa72edc"
server_key_iv: "db6a7b308191c93752422c17274eea39"
server_key_version: "encryptionkey@0"
I20260810 15:24:53.940412 28468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "113cba18140d4843b0df81e4ff428a5c"
format_stamp: "Formatted at 2026-08-10 15:24:53 on dist-test-slave-9zdj"
server_key: "881701b1613b332ada5751193aa72edc"
server_key_iv: "db6a7b308191c93752422c17274eea39"
server_key_version: "encryptionkey@0"
I20260810 15:24:53.942705 28468 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260810 15:24:53.943843 28485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:53.944082 28468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:53.944195 28468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "113cba18140d4843b0df81e4ff428a5c"
format_stamp: "Formatted at 2026-08-10 15:24:53 on dist-test-slave-9zdj"
server_key: "881701b1613b332ada5751193aa72edc"
server_key_iv: "db6a7b308191c93752422c17274eea39"
server_key_version: "encryptionkey@0"
I20260810 15:24:53.944289 28468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:53.970733 28468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:53.973373 28468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:53.973572 28468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:53.977734 28468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43471
I20260810 15:24:53.977782 28541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43471 every 8 connection(s)
I20260810 15:24:53.978199 28468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:53.979395 28542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:53.981840 28542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c: Bootstrap starting.
I20260810 15:24:53.982098 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28468
I20260810 15:24:53.982232 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:53.982467 4353 external_mini_cluster.cc:1428] Setting key a23d2b9b4b111900f07d7b33108d04f6
I20260810 15:24:53.982918 28542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:53.983618 28542 log.cc:824] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:53.984401 28542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c: No bootstrap required, opened a new log
I20260810 15:24:53.986428 28542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113cba18140d4843b0df81e4ff428a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260810 15:24:53.986575 28542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:53.986598 28542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113cba18140d4843b0df81e4ff428a5c, State: Initialized, Role: FOLLOWER
I20260810 15:24:53.986744 28542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113cba18140d4843b0df81e4ff428a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260810 15:24:53.986817 28542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:53.986840 28542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:53.986915 28542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:53.987493 28542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113cba18140d4843b0df81e4ff428a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260810 15:24:53.987624 28542 leader_election.cc:302] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 113cba18140d4843b0df81e4ff428a5c; no voters:
I20260810 15:24:53.987859 28542 leader_election.cc:288] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:53.987884 28549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:53.988082 28549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [term 1 LEADER]: Becoming Leader. State: Replica: 113cba18140d4843b0df81e4ff428a5c, State: Running, Role: LEADER
I20260810 15:24:53.988142 28542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:53.988193 28549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113cba18140d4843b0df81e4ff428a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } }
I20260810 15:24:53.988543 28549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 113cba18140d4843b0df81e4ff428a5c. Latest consensus state: current_term: 1 leader_uuid: "113cba18140d4843b0df81e4ff428a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113cba18140d4843b0df81e4ff428a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } } }
I20260810 15:24:53.988618 28549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:53.988780 28551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "113cba18140d4843b0df81e4ff428a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "113cba18140d4843b0df81e4ff428a5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43471 } } }
I20260810 15:24:53.988834 28551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:53.988960 28556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 15:24:53 dist-test-slave-9zdj krb5kdc[28454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:24:53.990415 28556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:53.992226 28556 catalog_manager.cc:1347] Generated new cluster ID: f078d0d9e6974ff28b4295fbf7e7ef8d
I20260810 15:24:53.992287 28556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:53.995106 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:53.983110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:32997 (local address 127.0.0.1:43471)
0810 15:24:53.983435 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:53.983444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:53.983473 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:24:53.984021 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:53.984028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:53.984084 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:53.984266 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:53.984977 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:53.986006 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:53.986935 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:53.987403 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:53.987537 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:53.990688 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:53.990692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:53.990692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:53.990705 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:53.993090 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:53.993555 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:53.993558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:53.993559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:53.993713 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:53.993954 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:53.993954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:53.993955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:53.994238 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:53.994703 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:53.994817 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 15:24:53.994916 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":204,"threads_started":1}
I20260810 15:24:53.996066 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:54.004971 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:53.997561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:37645 (local address 127.0.0.1:43471)
0810 15:24:53.997596 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:53.997597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:53.997759 (+ 162us) server_negotiation.cc:407] Connection header received
0810 15:24:53.997852 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:53.997853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:53.997861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:53.997887 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:53.998519 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:53.999266 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:54.000254 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:54.000556 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:54.000595 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:54.002137 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:54.002138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:54.002139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:54.002146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:54.003742 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:54.004155 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:54.004156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:54.004156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:54.004205 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:54.004401 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:54.004401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:54.004402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:54.004516 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:54.004585 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:54.004785 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 15:24:54.004858 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:24:54.010583 28556 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:54.011184 28556 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:54.022054 28556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 113cba18140d4843b0df81e4ff428a5c: Generated new TSK 0
I20260810 15:24:54.022289 28556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:54.025056 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28468
W20260810 15:24:54.035276 28451 connection.cc:535] client connection to 127.0.0.1:43471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43471): Connection reset by peer (error 104)
2026-08-10T15:24:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:54 dist-test-slave-9zdj krb5kdc[28579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:54 dist-test-slave-9zdj krb5kdc[28579](info): set up 2 sockets
Aug 10 15:24:54 dist-test-slave-9zdj krb5kdc[28579](info): commencing operation
krb5kdc: starting...
W20260810 15:24:55.162294 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:55 dist-test-slave-9zdj krb5kdc[28579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:55.305450 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:55.390960 28592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:55.391181 28592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:55.391206 28592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:55.392402 28592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:55.392452 28592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:55.392467 28592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:55.392480 28592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:55.392524 28592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:55.394160 28592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:55.394433 28592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:55.394676 28592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:55.399916 28601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:55.399960 28597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:55.399946 28599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:55.400238 28592 server_base.cc:1034] running on GCE node
I20260810 15:24:55.400502 28592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:55.400789 28592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:55.401969 28592 hybrid_clock.cc:648] HybridClock initialized: now 1786375495401946 us; error 39 us; skew 500 ppm
Aug 10 15:24:55 dist-test-slave-9zdj krb5kdc[28579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:55.404596 28592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:55.405094 28592 webserver.cc:460] Webserver started at http://127.0.0.1:37941/ using document root <none> and password file <none>
I20260810 15:24:55.405355 28592 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:55.405427 28592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:55.405582 28592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:55.406709 28592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e47047ede12a4a3889ebf1ee80b29361"
format_stamp: "Formatted at 2026-08-10 15:24:55 on dist-test-slave-9zdj"
server_key: "dec85060268f71d267496789e4c61e8a"
server_key_iv: "e604d6b1495451f9e0d0bf8d7afb7a6d"
server_key_version: "encryptionkey@0"
I20260810 15:24:55.407099 28592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e47047ede12a4a3889ebf1ee80b29361"
format_stamp: "Formatted at 2026-08-10 15:24:55 on dist-test-slave-9zdj"
server_key: "dec85060268f71d267496789e4c61e8a"
server_key_iv: "e604d6b1495451f9e0d0bf8d7afb7a6d"
server_key_version: "encryptionkey@0"
I20260810 15:24:55.408506 28592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:24:55.409274 28610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:55.409469 28592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:24:55.409593 28592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e47047ede12a4a3889ebf1ee80b29361"
format_stamp: "Formatted at 2026-08-10 15:24:55 on dist-test-slave-9zdj"
server_key: "dec85060268f71d267496789e4c61e8a"
server_key_iv: "e604d6b1495451f9e0d0bf8d7afb7a6d"
server_key_version: "encryptionkey@0"
I20260810 15:24:55.409730 28592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:55.434598 28592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:55.436821 28592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:55.437026 28592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:55.441874 28592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260810 15:24:55.441931 28668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260810 15:24:55.442343 28592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:55.443461 28670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:55.445986 28670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361: Bootstrap starting.
I20260810 15:24:55.446588 28670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:55.446848 28670 log.cc:824] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:55.447675 28670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361: No bootstrap required, opened a new log
I20260810 15:24:55.449306 28670 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47047ede12a4a3889ebf1ee80b29361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260810 15:24:55.449460 28670 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:55.449484 28670 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47047ede12a4a3889ebf1ee80b29361, State: Initialized, Role: FOLLOWER
I20260810 15:24:55.449677 28670 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47047ede12a4a3889ebf1ee80b29361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260810 15:24:55.449765 28670 raft_consensus.cc:388] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:55.449791 28670 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:55.449822 28670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:55.450371 28670 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47047ede12a4a3889ebf1ee80b29361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260810 15:24:55.450462 28670 leader_election.cc:302] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e47047ede12a4a3889ebf1ee80b29361; no voters:
I20260810 15:24:55.450603 28670 leader_election.cc:288] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:55.450695 28673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:55.450909 28673 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [term 1 LEADER]: Becoming Leader. State: Replica: e47047ede12a4a3889ebf1ee80b29361, State: Running, Role: LEADER
I20260810 15:24:55.450935 28670 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:55.451010 28673 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47047ede12a4a3889ebf1ee80b29361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260810 15:24:55.451416 28674 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e47047ede12a4a3889ebf1ee80b29361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47047ede12a4a3889ebf1ee80b29361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260810 15:24:55.451421 28675 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47047ede12a4a3889ebf1ee80b29361. Latest consensus state: current_term: 1 leader_uuid: "e47047ede12a4a3889ebf1ee80b29361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47047ede12a4a3889ebf1ee80b29361" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260810 15:24:55.451536 28674 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:55.451545 28675 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:55.451756 28677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:55.452073 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28592
I20260810 15:24:55.452194 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:55.452453 4353 external_mini_cluster.cc:1428] Setting key f4e27a4a0ca55bf84d634da3ceec34a0
I20260810 15:24:55.452682 28677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:55.454636 28677 catalog_manager.cc:1347] Generated new cluster ID: 73b90525769747038cecee35d3eff1f6
I20260810 15:24:55.454696 28677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:55.459234 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:55.453173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:45417 (local address 127.0.0.1:40977)
0810 15:24:55.453470 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:55.453480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:55.453517 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:24:55.453665 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:55.453672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:55.453736 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:55.453895 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:55.454170 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:55.455693 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:55.456853 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:55.457531 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:55.457677 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:55.458501 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:55.458506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:55.458506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:55.458732 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:55.458772 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:55.458787 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:55.458895 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:55.458974 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 15:24:55.459008 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260810 15:24:55.460677 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:55.461900 28677 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:55.462410 28677 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:55.469856 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:55.465972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:45959 (local address 127.0.0.1:40977)
0810 15:24:55.466043 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:55.466045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:55.466288 (+ 243us) server_negotiation.cc:407] Connection header received
0810 15:24:55.466349 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:55.466350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:55.466361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:55.466417 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:55.466544 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:55.467305 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:55.468399 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:55.468700 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:55.468770 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:55.469546 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:55.469547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:55.469547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:55.469552 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:55.469567 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:55.469571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:55.469669 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:55.469760 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 15:24:55.469772 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:24:55.473206 28677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47047ede12a4a3889ebf1ee80b29361: Generated new TSK 0
I20260810 15:24:55.473421 28677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:55.480567 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28592
W20260810 15:24:55.487872 28577 connection.cc:535] client connection to 127.0.0.1:40977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40977): Connection reset by peer (error 104)
2026-08-10T15:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:55 dist-test-slave-9zdj krb5kdc[28702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:55 dist-test-slave-9zdj krb5kdc[28702](info): set up 2 sockets
Aug 10 15:24:55 dist-test-slave-9zdj krb5kdc[28702](info): commencing operation
krb5kdc: starting...
W20260810 15:24:56.600888 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:56 dist-test-slave-9zdj krb5kdc[28702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:56.745141 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:56.831600 28720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:56.831836 28720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:56.831861 28720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:56.833583 28720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:56.833659 28720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:56.833675 28720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:56.833688 28720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:56.833701 28720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:56.835332 28720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:56.835678 28720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:56.835947 28720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:56.841197 28726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:56.841215 28725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:56.841215 28728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:56.841882 28720 server_base.cc:1034] running on GCE node
I20260810 15:24:56.842309 28720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:56.842708 28720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:56.843921 28720 hybrid_clock.cc:648] HybridClock initialized: now 1786375496843901 us; error 48 us; skew 500 ppm
Aug 10 15:24:56 dist-test-slave-9zdj krb5kdc[28702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:56.855970 28720 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:56.856609 28720 webserver.cc:460] Webserver started at http://127.0.0.1:42869/ using document root <none> and password file <none>
I20260810 15:24:56.856948 28720 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:56.857033 28720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:56.857213 28720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:56.858520 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "83cfde0ee1354f40a22ef982b6d841e3"
format_stamp: "Formatted at 2026-08-10 15:24:56 on dist-test-slave-9zdj"
server_key: "9cc779fa8deea1e689df773fe2154658"
server_key_iv: "d8add3f653d88827afa46e5e1486818b"
server_key_version: "encryptionkey@0"
I20260810 15:24:56.858942 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "83cfde0ee1354f40a22ef982b6d841e3"
format_stamp: "Formatted at 2026-08-10 15:24:56 on dist-test-slave-9zdj"
server_key: "9cc779fa8deea1e689df773fe2154658"
server_key_iv: "d8add3f653d88827afa46e5e1486818b"
server_key_version: "encryptionkey@0"
I20260810 15:24:56.860543 28720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:24:56.861545 28736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:56.861809 28720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:56.861927 28720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "83cfde0ee1354f40a22ef982b6d841e3"
format_stamp: "Formatted at 2026-08-10 15:24:56 on dist-test-slave-9zdj"
server_key: "9cc779fa8deea1e689df773fe2154658"
server_key_iv: "d8add3f653d88827afa46e5e1486818b"
server_key_version: "encryptionkey@0"
I20260810 15:24:56.862028 28720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:56.865950 28720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:56.868250 28720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:56.868424 28720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:56.872931 28720 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36939
I20260810 15:24:56.872967 28792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36939 every 8 connection(s)
I20260810 15:24:56.873342 28720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:56.874473 28793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:56.876791 28793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3: Bootstrap starting.
I20260810 15:24:56.877344 28793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:56.877578 28793 log.cc:824] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:56.878392 28793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3: No bootstrap required, opened a new log
I20260810 15:24:56.880016 28793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cfde0ee1354f40a22ef982b6d841e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260810 15:24:56.880170 28793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:56.880194 28793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cfde0ee1354f40a22ef982b6d841e3, State: Initialized, Role: FOLLOWER
I20260810 15:24:56.880308 28793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cfde0ee1354f40a22ef982b6d841e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260810 15:24:56.880401 28793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:56.880440 28793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:56.880497 28793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:56.881114 28793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cfde0ee1354f40a22ef982b6d841e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260810 15:24:56.881244 28793 leader_election.cc:302] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83cfde0ee1354f40a22ef982b6d841e3; no voters:
I20260810 15:24:56.881460 28793 leader_election.cc:288] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:56.881492 28797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:56.881639 28797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [term 1 LEADER]: Becoming Leader. State: Replica: 83cfde0ee1354f40a22ef982b6d841e3, State: Running, Role: LEADER
I20260810 15:24:56.881734 28797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cfde0ee1354f40a22ef982b6d841e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } }
I20260810 15:24:56.881798 28793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:56.882004 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28720
I20260810 15:24:56.882138 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:56.882339 28798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83cfde0ee1354f40a22ef982b6d841e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cfde0ee1354f40a22ef982b6d841e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } } }
I20260810 15:24:56.882397 4353 external_mini_cluster.cc:1428] Setting key b6ed53d0a7c48bcca3f55d15c83f6c72
I20260810 15:24:56.882438 28798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:56.882723 28809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:56.883460 28799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83cfde0ee1354f40a22ef982b6d841e3. Latest consensus state: current_term: 1 leader_uuid: "83cfde0ee1354f40a22ef982b6d841e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cfde0ee1354f40a22ef982b6d841e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36939 } } }
I20260810 15:24:56.883558 28799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:56.883692 28809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:56.885573 28809 catalog_manager.cc:1347] Generated new cluster ID: e5d2e4b9b29144ccbcf019aec7b3ba45
I20260810 15:24:56.885668 28809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:56.885797 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:56.883133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:60799 (local address 127.0.0.1:36939)
0810 15:24:56.883677 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:56.883688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:56.883894 (+ 206us) server_negotiation.cc:407] Connection header received
0810 15:24:56.884143 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:56.884151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:56.884182 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:56.884327 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:56.885161 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:56.885165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:56.885165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:56.885170 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:56.885225 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:56.885242 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:56.885323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:56.885412 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 15:24:56.885460 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":64,"threads_started":1}
I20260810 15:24:56.886812 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:56.889595 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:56.888381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:39521 (local address 127.0.0.1:36939)
0810 15:24:56.888417 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:56.888419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:56.888568 (+ 149us) server_negotiation.cc:407] Connection header received
0810 15:24:56.888598 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:56.888599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:56.888605 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:56.888685 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:56.889352 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:56.889354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:56.889354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:56.889359 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:56.889376 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:56.889381 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:56.889444 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:56.889497 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 15:24:56.889511 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:24:56.908810 28809 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:56.909045 28809 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:56.912292 28809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83cfde0ee1354f40a22ef982b6d841e3: Generated new TSK 0
I20260810 15:24:56.912511 28809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:56.920975 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28720
2026-08-10T15:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:56 dist-test-slave-9zdj krb5kdc[28832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:56 dist-test-slave-9zdj krb5kdc[28832](info): set up 2 sockets
Aug 10 15:24:56 dist-test-slave-9zdj krb5kdc[28832](info): commencing operation
krb5kdc: starting...
W20260810 15:24:58.078423 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:58 dist-test-slave-9zdj krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:58.222684 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:58.309988 28846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:58.310220 28846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:58.310245 28846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:58.311447 28846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:58.311497 28846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:58.311512 28846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:58.311525 28846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:58.311537 28846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:58.313024 28846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:58.313258 28846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:58.313450 28846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:58.319007 28855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:58.319199 28846 server_base.cc:1034] running on GCE node
W20260810 15:24:58.319007 28852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:58.319059 28851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:58.319669 28846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:58.319967 28846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:58.321115 28846 hybrid_clock.cc:648] HybridClock initialized: now 1786375498321095 us; error 43 us; skew 500 ppm
Aug 10 15:24:58 dist-test-slave-9zdj krb5kdc[28832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:58.323812 28846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:58.324324 28846 webserver.cc:460] Webserver started at http://127.0.0.1:38063/ using document root <none> and password file <none>
I20260810 15:24:58.324579 28846 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:58.324651 28846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:58.324817 28846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:58.326057 28846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e383d209315844c5a418a49ac76195fb"
format_stamp: "Formatted at 2026-08-10 15:24:58 on dist-test-slave-9zdj"
server_key: "4df4222570a06362324119da267f8152"
server_key_iv: "bb6d9b6284ba6dcf338d43498d323edc"
server_key_version: "encryptionkey@0"
I20260810 15:24:58.326491 28846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e383d209315844c5a418a49ac76195fb"
format_stamp: "Formatted at 2026-08-10 15:24:58 on dist-test-slave-9zdj"
server_key: "4df4222570a06362324119da267f8152"
server_key_iv: "bb6d9b6284ba6dcf338d43498d323edc"
server_key_version: "encryptionkey@0"
I20260810 15:24:58.327915 28846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:24:58.328718 28862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:58.328903 28846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:58.329011 28846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e383d209315844c5a418a49ac76195fb"
format_stamp: "Formatted at 2026-08-10 15:24:58 on dist-test-slave-9zdj"
server_key: "4df4222570a06362324119da267f8152"
server_key_iv: "bb6d9b6284ba6dcf338d43498d323edc"
server_key_version: "encryptionkey@0"
I20260810 15:24:58.329100 28846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:58.343468 28846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:58.345597 28846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:58.345832 28846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:58.350488 28846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35155
I20260810 15:24:58.350525 28921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35155 every 8 connection(s)
I20260810 15:24:58.350911 28846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:58.351909 28922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:58.354395 28922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb: Bootstrap starting.
I20260810 15:24:58.355028 28922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:58.355333 28922 log.cc:824] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:58.356125 28922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb: No bootstrap required, opened a new log
I20260810 15:24:58.357883 28922 raft_consensus.cc:348] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e383d209315844c5a418a49ac76195fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35155 } }
I20260810 15:24:58.358036 28922 raft_consensus.cc:374] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:58.358059 28922 raft_consensus.cc:729] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e383d209315844c5a418a49ac76195fb, State: Initialized, Role: FOLLOWER
I20260810 15:24:58.358224 28922 consensus_queue.cc:260] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e383d209315844c5a418a49ac76195fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35155 } }
I20260810 15:24:58.358291 28922 raft_consensus.cc:388] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:58.358315 28922 raft_consensus.cc:482] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:58.358394 28922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:58.359266 28922 raft_consensus.cc:504] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e383d209315844c5a418a49ac76195fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35155 } }
I20260810 15:24:58.359366 28922 leader_election.cc:302] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e383d209315844c5a418a49ac76195fb; no voters:
I20260810 15:24:58.359618 28922 leader_election.cc:288] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:58.359653 28926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:58.359818 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28846
I20260810 15:24:58.359889 28926 raft_consensus.cc:686] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [term 1 LEADER]: Becoming Leader. State: Replica: e383d209315844c5a418a49ac76195fb, State: Running, Role: LEADER
I20260810 15:24:58.359942 28922 sys_catalog.cc:564] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:58.359957 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:58.360006 28926 consensus_queue.cc:237] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e383d209315844c5a418a49ac76195fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35155 } }
I20260810 15:24:58.360194 4353 external_mini_cluster.cc:1428] Setting key 67de080f5a8a4948186b33f00c55ab78
I20260810 15:24:58.360525 28927 sys_catalog.cc:455] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e383d209315844c5a418a49ac76195fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e383d209315844c5a418a49ac76195fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35155 } } }
I20260810 15:24:58.360621 28927 sys_catalog.cc:458] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:58.360904 28928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e383d209315844c5a418a49ac76195fb. Latest consensus state: current_term: 1 leader_uuid: "e383d209315844c5a418a49ac76195fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e383d209315844c5a418a49ac76195fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35155 } } }
I20260810 15:24:58.361383 28936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:58.362069 28936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:58.361045 28928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:58.363873 28936 catalog_manager.cc:1347] Generated new cluster ID: 140204d527dd47e5ac5abceebfee244f
I20260810 15:24:58.363937 28936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:24:58 dist-test-slave-9zdj krb5kdc[28832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:24:58.372198 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:58.360898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:51289 (local address 127.0.0.1:35155)
0810 15:24:58.361495 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:58.361507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:58.361548 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:24:58.361820 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:58.361826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:58.361892 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:58.362063 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:58.362761 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:58.363944 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:58.364858 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:58.365348 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:58.365488 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:58.368074 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:58.368079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:58.368079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:58.368101 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:58.370426 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:58.370890 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:58.370894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:58.370894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:58.371004 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:58.371251 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:58.371252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:58.371252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:58.371540 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:58.371746 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:58.371855 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 15:24:58.371976 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":86,"threads_started":1}
I20260810 15:24:58.373425 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:58.384450 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:58.375030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40403 (local address 127.0.0.1:35155)
0810 15:24:58.375070 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:58.375072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:58.377815 (+ 2743us) server_negotiation.cc:407] Connection header received
0810 15:24:58.377902 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:58.377902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:58.377914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:58.377961 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:58.378581 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:58.379140 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:58.379944 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:58.380181 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:58.380263 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:58.381649 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:58.381650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:58.381651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:24:58.381659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:58.383187 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:58.383596 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:58.383598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:58.383599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:58.383662 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:24:58.383898 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:24:58.383899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:24:58.383900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:24:58.384020 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:58.384134 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:58.384265 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 15:24:58.384333 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260810 15:24:58.410835 28936 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:58.411695 28936 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:58.420465 28936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e383d209315844c5a418a49ac76195fb: Generated new TSK 0
I20260810 15:24:58.420770 28936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:58.424268 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28846
W20260810 15:24:58.432631 28830 connection.cc:535] client connection to 127.0.0.1:35155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35155): Connection reset by peer (error 104)
2026-08-10T15:24:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:24:58 dist-test-slave-9zdj krb5kdc[28957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:24:58 dist-test-slave-9zdj krb5kdc[28957](info): set up 2 sockets
Aug 10 15:24:58 dist-test-slave-9zdj krb5kdc[28957](info): commencing operation
krb5kdc: starting...
W20260810 15:24:59.627830 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:24:59 dist-test-slave-9zdj krb5kdc[28957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:24:59.768761 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:24:59.860146 28972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:24:59.860400 28972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:24:59.860424 28972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:24:59.861547 28972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:24:59.861586 28972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:24:59.861600 28972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:24:59.861641 28972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:24:59.861655 28972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:24:59.863457 28972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:24:59.863706 28972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:24:59.863898 28972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:24:59.869400 28978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:24:59.869392 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:59.869424 28972 server_base.cc:1034] running on GCE node
W20260810 15:24:59.869366 28977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:24:59.870015 28972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:24:59.870374 28972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:24:59.871539 28972 hybrid_clock.cc:648] HybridClock initialized: now 1786375499871502 us; error 53 us; skew 500 ppm
Aug 10 15:24:59 dist-test-slave-9zdj krb5kdc[28957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:24:59.874619 28972 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:24:59.875252 28972 webserver.cc:460] Webserver started at http://127.0.0.1:34347/ using document root <none> and password file <none>
I20260810 15:24:59.875501 28972 fs_manager.cc:359] Metadata directory not provided
I20260810 15:24:59.875548 28972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:24:59.875710 28972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:24:59.877180 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "4b2e642b9c5642cfa842656b59cb09a4"
format_stamp: "Formatted at 2026-08-10 15:24:59 on dist-test-slave-9zdj"
server_key: "c17c25d4977eb813585c5ad486510d8f"
server_key_iv: "4ec4ea3cbafad1eaf011c16a6757b080"
server_key_version: "encryptionkey@0"
I20260810 15:24:59.877570 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "4b2e642b9c5642cfa842656b59cb09a4"
format_stamp: "Formatted at 2026-08-10 15:24:59 on dist-test-slave-9zdj"
server_key: "c17c25d4977eb813585c5ad486510d8f"
server_key_iv: "4ec4ea3cbafad1eaf011c16a6757b080"
server_key_version: "encryptionkey@0"
I20260810 15:24:59.879364 28972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:59.880247 28987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:24:59.880476 28972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:24:59.880584 28972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "4b2e642b9c5642cfa842656b59cb09a4"
format_stamp: "Formatted at 2026-08-10 15:24:59 on dist-test-slave-9zdj"
server_key: "c17c25d4977eb813585c5ad486510d8f"
server_key_iv: "4ec4ea3cbafad1eaf011c16a6757b080"
server_key_version: "encryptionkey@0"
I20260810 15:24:59.880662 28972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:24:59.890319 28972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:24:59.892885 28972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:24:59.893059 28972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:24:59.897410 28972 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34339
I20260810 15:24:59.897493 29044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34339 every 8 connection(s)
I20260810 15:24:59.897898 28972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:24:59.898869 29045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:24:59.901263 29045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4: Bootstrap starting.
I20260810 15:24:59.901947 29045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4: Neither blocks nor log segments found. Creating new log.
I20260810 15:24:59.902278 29045 log.cc:824] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:24:59.903081 29045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4: No bootstrap required, opened a new log
I20260810 15:24:59.904778 29045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e642b9c5642cfa842656b59cb09a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 15:24:59.904954 29045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:24:59.905007 29045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2e642b9c5642cfa842656b59cb09a4, State: Initialized, Role: FOLLOWER
I20260810 15:24:59.905210 29045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e642b9c5642cfa842656b59cb09a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 15:24:59.905315 29045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:24:59.905361 29045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:24:59.905412 29045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:24:59.905587 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 28972
I20260810 15:24:59.905776 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:24:59.906009 4353 external_mini_cluster.cc:1428] Setting key eb560ffebd549239727670feac7b27a5
I20260810 15:24:59.906216 29045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e642b9c5642cfa842656b59cb09a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 15:24:59.906352 29045 leader_election.cc:302] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2e642b9c5642cfa842656b59cb09a4; no voters:
I20260810 15:24:59.906549 29045 leader_election.cc:288] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:24:59.906634 29048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:24:59.906883 29048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4b2e642b9c5642cfa842656b59cb09a4, State: Running, Role: LEADER
I20260810 15:24:59.906936 29045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:24:59.907064 29048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e642b9c5642cfa842656b59cb09a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } }
I20260810 15:24:59.907768 29050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2e642b9c5642cfa842656b59cb09a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e642b9c5642cfa842656b59cb09a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } } }
I20260810 15:24:59.907893 29050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:59.907917 29052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2e642b9c5642cfa842656b59cb09a4. Latest consensus state: current_term: 1 leader_uuid: "4b2e642b9c5642cfa842656b59cb09a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2e642b9c5642cfa842656b59cb09a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34339 } } }
I20260810 15:24:59.907990 29052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4 [sys.catalog]: This master's current role is: LEADER
I20260810 15:24:59.908277 29061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:24:59.908805 29061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:24:59.910570 29061 catalog_manager.cc:1347] Generated new cluster ID: 91999b5aee0d438c82afe55cc0befb14
I20260810 15:24:59.910626 29061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:24:59.913028 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:59.906763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:60471 (local address 127.0.0.1:34339)
0810 15:24:59.906997 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:59.907008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:59.907044 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:24:59.907188 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:59.907196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:59.907258 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:59.907469 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:59.907692 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:59.909356 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:59.910513 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:59.911146 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:59.911621 (+ 475us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:59.912115 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:59.912119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:59.912120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:59.912333 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:59.912373 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:59.912389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:59.912714 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:59.912758 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:24:59.912801 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260810 15:24:59.914310 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:24:59.920058 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:24:59.916159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40945 (local address 127.0.0.1:34339)
0810 15:24:59.916197 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:24:59.916199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:24:59.916260 (+ 61us) server_negotiation.cc:407] Connection header received
0810 15:24:59.916330 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:24:59.916331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:24:59.916532 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:24:59.916602 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:24:59.916774 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:59.917537 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:59.918613 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:24:59.918948 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:24:59.919030 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:24:59.919786 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:24:59.919787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:24:59.919787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:24:59.919792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:24:59.919809 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:24:59.919814 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:24:59.919882 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:24:59.919955 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 15:24:59.919968 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:24:59.949122 29061 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:24:59.949967 29061 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:24:59.954538 29061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b2e642b9c5642cfa842656b59cb09a4: Generated new TSK 0
I20260810 15:24:59.954802 29061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:24:59.959488 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 28972
W20260810 15:24:59.969499 28955 connection.cc:535] client connection to 127.0.0.1:34339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34339): Connection reset by peer (error 104)
2026-08-10T15:24:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:00 dist-test-slave-9zdj krb5kdc[29081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:00 dist-test-slave-9zdj krb5kdc[29081](info): set up 2 sockets
Aug 10 15:25:00 dist-test-slave-9zdj krb5kdc[29081](info): commencing operation
krb5kdc: starting...
W20260810 15:25:01.120939 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:01 dist-test-slave-9zdj krb5kdc[29081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:25:01.259332 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:01.344313 29094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:01.344553 29094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:01.344578 29094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:01.345881 29094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:01.345934 29094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:01.345949 29094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:01.345968 29094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:01.345981 29094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:01.347502 29094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:01.347725 29094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:01.347923 29094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:01.353101 29104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:01.353112 29102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:01.353171 29094 server_base.cc:1034] running on GCE node
W20260810 15:25:01.353101 29099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:01.353731 29094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:01.354074 29094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:01.355228 29094 hybrid_clock.cc:648] HybridClock initialized: now 1786375501355213 us; error 34 us; skew 500 ppm
Aug 10 15:25:01 dist-test-slave-9zdj krb5kdc[29081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:01.357996 29094 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:25:01.358530 29094 webserver.cc:460] Webserver started at http://127.0.0.1:41543/ using document root <none> and password file <none>
I20260810 15:25:01.358824 29094 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:01.358875 29094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:01.359048 29094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:01.360152 29094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "afeaefdd7463453892936910e68f09b3"
format_stamp: "Formatted at 2026-08-10 15:25:01 on dist-test-slave-9zdj"
server_key: "c78859487ce1211158aa1b03404fb640"
server_key_iv: "21e0b16cace0b891980e741c4d4cb8e9"
server_key_version: "encryptionkey@0"
I20260810 15:25:01.360581 29094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "afeaefdd7463453892936910e68f09b3"
format_stamp: "Formatted at 2026-08-10 15:25:01 on dist-test-slave-9zdj"
server_key: "c78859487ce1211158aa1b03404fb640"
server_key_iv: "21e0b16cace0b891980e741c4d4cb8e9"
server_key_version: "encryptionkey@0"
I20260810 15:25:01.362075 29094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:25:01.362968 29112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:01.363183 29094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:01.363337 29094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "afeaefdd7463453892936910e68f09b3"
format_stamp: "Formatted at 2026-08-10 15:25:01 on dist-test-slave-9zdj"
server_key: "c78859487ce1211158aa1b03404fb640"
server_key_iv: "21e0b16cace0b891980e741c4d4cb8e9"
server_key_version: "encryptionkey@0"
I20260810 15:25:01.363425 29094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:01.367118 29094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:01.369215 29094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:01.369401 29094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:01.373525 29094 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42195
I20260810 15:25:01.373579 29170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42195 every 8 connection(s)
I20260810 15:25:01.373970 29094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:01.375051 29171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:01.375833 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29094
I20260810 15:25:01.375962 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:01.376160 4353 external_mini_cluster.cc:1428] Setting key eda2736256cb0b3b728031296a659c6a
I20260810 15:25:01.378453 29171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3: Bootstrap starting.
I20260810 15:25:01.378767 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:01.376754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40071 (local address 127.0.0.1:42195)
0810 15:25:01.376937 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:01.376947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:01.376983 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:25:01.377384 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:01.377391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:01.377414 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:01.377529 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:01.378272 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:01.378276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:01.378276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:01.378281 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:01.378323 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:01.378333 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:01.378392 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:01.378483 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 15:25:01.378543 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260810 15:25:01.379247 29171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:01.379622 29171 log.cc:824] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:01.380055 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:01.380445 29171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3: No bootstrap required, opened a new log
I20260810 15:25:01.382985 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:01.381931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40521 (local address 127.0.0.1:42195)
0810 15:25:01.381976 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:01.381978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:01.382091 (+ 113us) server_negotiation.cc:407] Connection header received
0810 15:25:01.382154 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:01.382155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:01.382159 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:01.382207 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:01.382783 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:01.382784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:01.382785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:01.382788 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:01.382801 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:01.382804 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:01.382847 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:01.382910 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 15:25:01.382924 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:25:01.383302 29171 raft_consensus.cc:348] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afeaefdd7463453892936910e68f09b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260810 15:25:01.383479 29171 raft_consensus.cc:374] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:01.383533 29171 raft_consensus.cc:729] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afeaefdd7463453892936910e68f09b3, State: Initialized, Role: FOLLOWER
I20260810 15:25:01.383662 29171 consensus_queue.cc:260] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afeaefdd7463453892936910e68f09b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260810 15:25:01.383746 29171 raft_consensus.cc:388] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:01.383770 29171 raft_consensus.cc:482] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:01.383838 29171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:01.384433 29171 raft_consensus.cc:504] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afeaefdd7463453892936910e68f09b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260810 15:25:01.384550 29171 leader_election.cc:302] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afeaefdd7463453892936910e68f09b3; no voters:
I20260810 15:25:01.384712 29171 leader_election.cc:288] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:01.384752 29182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:01.384913 29182 raft_consensus.cc:686] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [term 1 LEADER]: Becoming Leader. State: Replica: afeaefdd7463453892936910e68f09b3, State: Running, Role: LEADER
I20260810 15:25:01.385027 29171 sys_catalog.cc:564] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:01.385035 29182 consensus_queue.cc:237] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afeaefdd7463453892936910e68f09b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260810 15:25:01.385413 29182 sys_catalog.cc:455] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afeaefdd7463453892936910e68f09b3. Latest consensus state: current_term: 1 leader_uuid: "afeaefdd7463453892936910e68f09b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afeaefdd7463453892936910e68f09b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } } }
I20260810 15:25:01.385425 29184 sys_catalog.cc:455] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afeaefdd7463453892936910e68f09b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afeaefdd7463453892936910e68f09b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } } }
I20260810 15:25:01.385581 29182 sys_catalog.cc:458] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:01.385855 29184 sys_catalog.cc:458] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:01.386082 29190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:01.386585 29190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:01.388186 29190 catalog_manager.cc:1347] Generated new cluster ID: 551f70bb4c494e258b33f896bba7e695
I20260810 15:25:01.388304 29190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:01.402314 29190 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:01.402581 29190 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:01.407181 29190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afeaefdd7463453892936910e68f09b3: Generated new TSK 0
I20260810 15:25:01.407377 29190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:01.413981 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29094
2026-08-10T15:25:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:01 dist-test-slave-9zdj krb5kdc[29204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:01 dist-test-slave-9zdj krb5kdc[29204](info): set up 2 sockets
Aug 10 15:25:01 dist-test-slave-9zdj krb5kdc[29204](info): commencing operation
krb5kdc: starting...
W20260810 15:25:02.596859 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:02 dist-test-slave-9zdj krb5kdc[29204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:02.733949 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:42703
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:42703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:02.828193 29220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:02.828451 29220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:02.828478 29220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:02.829787 29220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:02.829839 29220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:02.829855 29220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:02.829869 29220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:02.829883 29220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:02.831611 29220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:42703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:42703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:02.831851 29220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:02.832055 29220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:02.837819 29229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:02.837819 29227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:02.837899 29226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:02.838236 29220 server_base.cc:1034] running on GCE node
I20260810 15:25:02.838542 29220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:02.838881 29220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:02.840065 29220 hybrid_clock.cc:648] HybridClock initialized: now 1786375502839917 us; error 167 us; skew 500 ppm
Aug 10 15:25:02 dist-test-slave-9zdj krb5kdc[29204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375502, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:02.847994 29220 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:02.848582 29220 webserver.cc:460] Webserver started at http://127.4.64.126:45599/ using document root <none> and password file <none>
I20260810 15:25:02.848827 29220 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:02.848902 29220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:02.849066 29220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:02.850278 29220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "ebb70603a28c452f96ea41759714033a"
format_stamp: "Formatted at 2026-08-10 15:25:02 on dist-test-slave-9zdj"
server_key: "f8c5e943060921cd052c626c036d71df"
server_key_iv: "de9cc976bd2ddaa0f7de5c8466cb0123"
server_key_version: "encryptionkey@0"
I20260810 15:25:02.850690 29220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "ebb70603a28c452f96ea41759714033a"
format_stamp: "Formatted at 2026-08-10 15:25:02 on dist-test-slave-9zdj"
server_key: "f8c5e943060921cd052c626c036d71df"
server_key_iv: "de9cc976bd2ddaa0f7de5c8466cb0123"
server_key_version: "encryptionkey@0"
I20260810 15:25:02.852195 29220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:25:02.853011 29237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:02.853255 29220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:02.853379 29220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "ebb70603a28c452f96ea41759714033a"
format_stamp: "Formatted at 2026-08-10 15:25:02 on dist-test-slave-9zdj"
server_key: "f8c5e943060921cd052c626c036d71df"
server_key_iv: "de9cc976bd2ddaa0f7de5c8466cb0123"
server_key_version: "encryptionkey@0"
I20260810 15:25:02.853461 29220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:02.864776 29220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:02.867331 29220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:02.867524 29220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:02.871948 29220 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:42703
I20260810 15:25:02.871986 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:42703 every 8 connection(s)
I20260810 15:25:02.872393 29220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:02.873351 29293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:02.875722 29293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a: Bootstrap starting.
I20260810 15:25:02.876288 29293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:02.876542 29293 log.cc:824] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:02.877249 29293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a: No bootstrap required, opened a new log
I20260810 15:25:02.878875 29293 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb70603a28c452f96ea41759714033a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42703 } }
I20260810 15:25:02.879022 29293 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:02.879063 29293 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb70603a28c452f96ea41759714033a, State: Initialized, Role: FOLLOWER
I20260810 15:25:02.879216 29293 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb70603a28c452f96ea41759714033a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42703 } }
I20260810 15:25:02.879299 29293 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:02.879323 29293 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:02.879352 29293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:02.880187 29293 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb70603a28c452f96ea41759714033a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42703 } }
I20260810 15:25:02.880282 29293 leader_election.cc:302] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebb70603a28c452f96ea41759714033a; no voters:
I20260810 15:25:02.880409 29293 leader_election.cc:288] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:02.880484 29296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:02.880658 29296 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [term 1 LEADER]: Becoming Leader. State: Replica: ebb70603a28c452f96ea41759714033a, State: Running, Role: LEADER
I20260810 15:25:02.880712 29293 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:02.880761 29296 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb70603a28c452f96ea41759714033a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42703 } }
I20260810 15:25:02.881116 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29220
I20260810 15:25:02.881274 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:02.881510 4353 external_mini_cluster.cc:1428] Setting key d2efc3692c230be72f06484629475bf5
I20260810 15:25:02.881513 29298 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebb70603a28c452f96ea41759714033a. Latest consensus state: current_term: 1 leader_uuid: "ebb70603a28c452f96ea41759714033a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb70603a28c452f96ea41759714033a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42703 } } }
I20260810 15:25:02.881589 29298 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:02.881834 29297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebb70603a28c452f96ea41759714033a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebb70603a28c452f96ea41759714033a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 42703 } } }
I20260810 15:25:02.881944 29297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:02.881901 29308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:02.883126 29308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:02.884761 29308 catalog_manager.cc:1347] Generated new cluster ID: c24aa737c1c74caa93031a4f615f58b0
I20260810 15:25:02.884819 29308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:25:02 dist-test-slave-9zdj krb5kdc[29204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:25:02.892609 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:02.882069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41917 (local address 127.4.64.126:42703)
0810 15:25:02.882482 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:02.882493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:02.882532 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:25:02.882676 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:02.882683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:02.882742 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:02.882924 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:02.883774 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:02.884919 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:02.885801 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:02.886230 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:02.886359 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:02.888981 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:02.888986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:02.888986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:02.889001 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:02.890968 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:02.891390 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:02.891394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:02.891394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:02.891504 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:02.891757 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:02.891758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:02.891759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:02.892013 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:02.892223 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:02.892312 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 15:25:02.892420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":64,"threads_started":1}
I20260810 15:25:02.893647 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:02.894848 29308 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:02.895699 29308 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:02.901569 29308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebb70603a28c452f96ea41759714033a: Generated new TSK 0
I20260810 15:25:02.901793 29308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:02.904699 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:02.897962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38553 (local address 127.4.64.126:42703)
0810 15:25:02.898005 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:02.898007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:02.898138 (+ 131us) server_negotiation.cc:407] Connection header received
0810 15:25:02.898220 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:02.898221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:02.898232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:02.898277 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:02.898822 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:02.899405 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:02.900167 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:02.900391 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:02.900472 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:02.902019 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:02.902021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:02.902021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:02.902029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:02.903460 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:02.903845 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:02.903847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:02.903847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:02.903908 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:02.904146 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:02.904147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:02.904148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:02.904281 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:02.904400 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:02.904526 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 15:25:02.904590 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:25:02.907094 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29220
W20260810 15:25:02.916065 29202 connection.cc:535] client connection to 127.4.64.126:42703 recv error: Network error: failed to read from TLS socket (remote: 127.4.64.126:42703): Connection reset by peer (error 104)
2026-08-10T15:25:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1493 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:02 dist-test-slave-9zdj krb5kdc[29328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:02 dist-test-slave-9zdj krb5kdc[29328](info): set up 2 sockets
Aug 10 15:25:02 dist-test-slave-9zdj krb5kdc[29328](info): commencing operation
krb5kdc: starting...
W20260810 15:25:04.039505 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:04 dist-test-slave-9zdj krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:04.180606 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43243
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:04.266882 29343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:04.267158 29343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:04.267197 29343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:04.268380 29343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:04.268464 29343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:04.268488 29343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:04.268508 29343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:04.268538 29343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:04.270061 29343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:04.270377 29343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:04.270628 29343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:04.275796 29348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:04.275841 29349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:04.275954 29343 server_base.cc:1034] running on GCE node
W20260810 15:25:04.276026 29351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:04.276466 29343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:04.276806 29343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:04.277990 29343 hybrid_clock.cc:648] HybridClock initialized: now 1786375504277974 us; error 47 us; skew 500 ppm
Aug 10 15:25:04 dist-test-slave-9zdj krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375504, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:04.280808 29343 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:04.281307 29343 webserver.cc:460] Webserver started at http://127.4.64.126:46827/ using document root <none> and password file <none>
I20260810 15:25:04.281584 29343 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:04.281693 29343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:04.281862 29343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:04.282982 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "4fe080b5fc1f429ebdd4233fd31204ba"
format_stamp: "Formatted at 2026-08-10 15:25:04 on dist-test-slave-9zdj"
server_key: "2a7bd7d4c09cb8af8dd74a3bfde14ed3"
server_key_iv: "1012cae5e402d0b5b39ccd3994183fc3"
server_key_version: "encryptionkey@0"
I20260810 15:25:04.283422 29343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "4fe080b5fc1f429ebdd4233fd31204ba"
format_stamp: "Formatted at 2026-08-10 15:25:04 on dist-test-slave-9zdj"
server_key: "2a7bd7d4c09cb8af8dd74a3bfde14ed3"
server_key_iv: "1012cae5e402d0b5b39ccd3994183fc3"
server_key_version: "encryptionkey@0"
I20260810 15:25:04.284878 29343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:25:04.285759 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:04.285939 29343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:25:04.286038 29343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "4fe080b5fc1f429ebdd4233fd31204ba"
format_stamp: "Formatted at 2026-08-10 15:25:04 on dist-test-slave-9zdj"
server_key: "2a7bd7d4c09cb8af8dd74a3bfde14ed3"
server_key_iv: "1012cae5e402d0b5b39ccd3994183fc3"
server_key_version: "encryptionkey@0"
I20260810 15:25:04.286123 29343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:04.311851 29343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:04.314581 29343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:04.314765 29343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:04.319262 29343 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:43243
I20260810 15:25:04.319479 29415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:43243 every 8 connection(s)
I20260810 15:25:04.319789 29343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:04.321107 29416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:04.324787 29416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba: Bootstrap starting.
I20260810 15:25:04.325592 29416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:04.325956 29416 log.cc:824] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:04.326820 29416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba: No bootstrap required, opened a new log
I20260810 15:25:04.327211 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29343
I20260810 15:25:04.327366 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:04.327575 4353 external_mini_cluster.cc:1428] Setting key 0051fdfeeab69285a7fd6011d7cb64f9
I20260810 15:25:04.329697 29416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43243 } }
I20260810 15:25:04.329952 29416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:04.330032 29416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe080b5fc1f429ebdd4233fd31204ba, State: Initialized, Role: FOLLOWER
I20260810 15:25:04.330219 29416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43243 } }
I20260810 15:25:04.330400 29416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:04.330458 29416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:04.330508 29416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:04.331409 29416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43243 } }
I20260810 15:25:04.331543 29416 leader_election.cc:302] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe080b5fc1f429ebdd4233fd31204ba; no voters:
I20260810 15:25:04.331727 29416 leader_election.cc:288] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:04.331764 29421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:04.331877 29421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [term 1 LEADER]: Becoming Leader. State: Replica: 4fe080b5fc1f429ebdd4233fd31204ba, State: Running, Role: LEADER
I20260810 15:25:04.331964 29421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43243 } }
I20260810 15:25:04.332026 29416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:04.332357 29423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43243 } } }
I20260810 15:25:04.332507 29423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:04.332856 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe080b5fc1f429ebdd4233fd31204ba. Latest consensus state: current_term: 1 leader_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe080b5fc1f429ebdd4233fd31204ba" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43243 } } }
I20260810 15:25:04.332979 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:04.332907 29429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:04.333678 29429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:04.335758 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:04.328162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33313 (local address 127.4.64.126:43243)
0810 15:25:04.328406 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:04.328418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:04.328455 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:25:04.329705 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:04.329722 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:04.329784 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:04.330011 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:04.330335 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:04.331851 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:04.333052 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:04.333722 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:04.334826 (+ 1104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:04.334991 (+ 165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:04.334994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:04.334994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:04.335182 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:04.335214 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:04.335224 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:04.335330 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:04.335557 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 15:25:04.335582 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20260810 15:25:04.336753 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:04.341096 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:04.338189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44713 (local address 127.4.64.126:43243)
0810 15:25:04.338226 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:04.338228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:04.338360 (+ 132us) server_negotiation.cc:407] Connection header received
0810 15:25:04.338433 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:04.338434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:04.338445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:04.338491 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:04.338617 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:04.339107 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:04.339835 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:04.340065 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:04.340163 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:04.340839 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:04.340840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:04.340840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:04.340845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:04.340859 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:04.340863 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:04.340917 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:04.340977 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 15:25:04.340989 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:25:04.342896 29429 catalog_manager.cc:1347] Generated new cluster ID: 9d8543cbe8e1484fb25ed3124ec4b3bf
I20260810 15:25:04.342952 29429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:04.348632 29429 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:04.349157 29429 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:04.351909 29429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fe080b5fc1f429ebdd4233fd31204ba: Generated new TSK 0
I20260810 15:25:04.352099 29429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:04.355991 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29343
W20260810 15:25:04.367084 29326 connection.cc:535] client connection to 127.4.64.126:43243 recv error: Network error: failed to read from TLS socket (remote: 127.4.64.126:43243): Connection reset by peer (error 104)
2026-08-10T15:25:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:04 dist-test-slave-9zdj krb5kdc[29449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:04 dist-test-slave-9zdj krb5kdc[29449](info): set up 2 sockets
Aug 10 15:25:04 dist-test-slave-9zdj krb5kdc[29449](info): commencing operation
krb5kdc: starting...
W20260810 15:25:05.524358 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:05 dist-test-slave-9zdj krb5kdc[29449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:05.670830 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:41091
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:41091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:05.756470 29462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:05.756692 29462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:05.756716 29462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:05.757874 29462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:05.757918 29462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:05.757932 29462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:05.757946 29462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:05.757957 29462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:05.759384 29462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:41091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:41091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:05.759591 29462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:05.759776 29462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:05.765009 29471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:05.765009 29467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:05.765035 29468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:05.765156 29462 server_base.cc:1034] running on GCE node
I20260810 15:25:05.765650 29462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:05.765947 29462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:05.767144 29462 hybrid_clock.cc:648] HybridClock initialized: now 1786375505767129 us; error 30 us; skew 500 ppm
Aug 10 15:25:05 dist-test-slave-9zdj krb5kdc[29449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375505, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:05.769843 29462 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:05.770329 29462 webserver.cc:460] Webserver started at http://127.4.64.126:35973/ using document root <none> and password file <none>
I20260810 15:25:05.770524 29462 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:05.770565 29462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:05.770666 29462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:05.771730 29462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "4ea6139879214a2081802ace0915e2e5"
format_stamp: "Formatted at 2026-08-10 15:25:05 on dist-test-slave-9zdj"
server_key: "639ad62d2036d012a38a68e4af93ecbe"
server_key_iv: "9efa96dc8d259a3e8bf58999f214d6bf"
server_key_version: "encryptionkey@0"
I20260810 15:25:05.772042 29462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "4ea6139879214a2081802ace0915e2e5"
format_stamp: "Formatted at 2026-08-10 15:25:05 on dist-test-slave-9zdj"
server_key: "639ad62d2036d012a38a68e4af93ecbe"
server_key_iv: "9efa96dc8d259a3e8bf58999f214d6bf"
server_key_version: "encryptionkey@0"
I20260810 15:25:05.773413 29462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:25:05.774294 29480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:05.774524 29462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:25:05.774629 29462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "4ea6139879214a2081802ace0915e2e5"
format_stamp: "Formatted at 2026-08-10 15:25:05 on dist-test-slave-9zdj"
server_key: "639ad62d2036d012a38a68e4af93ecbe"
server_key_iv: "9efa96dc8d259a3e8bf58999f214d6bf"
server_key_version: "encryptionkey@0"
I20260810 15:25:05.774720 29462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:05.778148 29462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:05.780180 29462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:05.780299 29462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:05.784555 29462 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:41091
I20260810 15:25:05.784603 29537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:41091 every 8 connection(s)
I20260810 15:25:05.785012 29462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:05.786159 29538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:05.787115 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29462
I20260810 15:25:05.787237 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:05.787467 4353 external_mini_cluster.cc:1428] Setting key 49b0fc070a1cfa3889a042ce85b9c694
I20260810 15:25:05.789871 29538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5: Bootstrap starting.
I20260810 15:25:05.790115 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:05.788062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51017 (local address 127.4.64.126:41091)
0810 15:25:05.788248 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:05.788256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:05.788283 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:25:05.788700 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:05.788706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:05.788730 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:05.788847 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:05.789578 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:05.789581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:05.789582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:05.789585 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:05.789672 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:05.789689 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:05.789761 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:05.789849 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 15:25:05.789904 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260810 15:25:05.790661 29538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:05.791076 29538 log.cc:824] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:05.791167 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:05.791851 29538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5: No bootstrap required, opened a new log
I20260810 15:25:05.793757 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:05.792669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49463 (local address 127.4.64.126:41091)
0810 15:25:05.792711 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:05.792713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:05.792796 (+ 83us) server_negotiation.cc:407] Connection header received
0810 15:25:05.792880 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:05.792881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:05.792885 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:05.792931 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:05.793488 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:05.793489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:05.793489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:05.793493 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:05.793505 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:05.793509 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:05.793556 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:05.793667 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 15:25:05.793681 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:25:05.794864 29538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea6139879214a2081802ace0915e2e5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41091 } }
I20260810 15:25:05.795051 29538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:05.795099 29538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea6139879214a2081802ace0915e2e5, State: Initialized, Role: FOLLOWER
I20260810 15:25:05.795239 29538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea6139879214a2081802ace0915e2e5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41091 } }
I20260810 15:25:05.795341 29538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:05.795387 29538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:05.795441 29538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:05.796070 29538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea6139879214a2081802ace0915e2e5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41091 } }
I20260810 15:25:05.796195 29538 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea6139879214a2081802ace0915e2e5; no voters:
I20260810 15:25:05.796422 29538 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:05.796442 29548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:05.796603 29548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea6139879214a2081802ace0915e2e5, State: Running, Role: LEADER
I20260810 15:25:05.796693 29538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:05.796784 29548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea6139879214a2081802ace0915e2e5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41091 } }
I20260810 15:25:05.797165 29550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea6139879214a2081802ace0915e2e5. Latest consensus state: current_term: 1 leader_uuid: "4ea6139879214a2081802ace0915e2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea6139879214a2081802ace0915e2e5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41091 } } }
I20260810 15:25:05.797158 29549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea6139879214a2081802ace0915e2e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea6139879214a2081802ace0915e2e5" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 41091 } } }
I20260810 15:25:05.797302 29550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:05.797369 29549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:05.797664 29558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:05.798247 29558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:05.799798 29558 catalog_manager.cc:1347] Generated new cluster ID: 5d0f5393d64a454896a19fab0a3b5618
I20260810 15:25:05.799858 29558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:05.809870 29558 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:05.810098 29558 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:05.816846 29558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea6139879214a2081802ace0915e2e5: Generated new TSK 0
I20260810 15:25:05.817070 29558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:05.819885 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29462
2026-08-10T15:25:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1460 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:05 dist-test-slave-9zdj krb5kdc[29573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:05 dist-test-slave-9zdj krb5kdc[29573](info): set up 2 sockets
Aug 10 15:25:05 dist-test-slave-9zdj krb5kdc[29573](info): commencing operation
krb5kdc: starting...
W20260810 15:25:06.953436 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:07 dist-test-slave-9zdj krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:07.093044 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44655
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:44655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:07.177790 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:07.178006 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:07.178030 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:07.179201 29589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:07.179246 29589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:07.179261 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:07.179275 29589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:07.179327 29589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:07.180791 29589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:44655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:07.181054 29589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:07.181303 29589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:07.186512 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:07.186942 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:07.186942 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:07.187351 29589 server_base.cc:1034] running on GCE node
I20260810 15:25:07.187666 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:07.188023 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:07.189206 29589 hybrid_clock.cc:648] HybridClock initialized: now 1786375507189227 us; error 72 us; skew 500 ppm
Aug 10 15:25:07 dist-test-slave-9zdj krb5kdc[29573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:07.192018 29589 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:07.192560 29589 webserver.cc:460] Webserver started at http://127.4.64.126:34969/ using document root <none> and password file <none>
I20260810 15:25:07.192811 29589 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:07.192883 29589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:07.193040 29589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:07.194278 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e25d90a07f814a7a8159077ce022a01c"
format_stamp: "Formatted at 2026-08-10 15:25:07 on dist-test-slave-9zdj"
server_key: "1796af53025531cf53c8d01d16c7d13c"
server_key_iv: "0e99de91c171b3a84df3044f818ee6b5"
server_key_version: "encryptionkey@0"
I20260810 15:25:07.194670 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e25d90a07f814a7a8159077ce022a01c"
format_stamp: "Formatted at 2026-08-10 15:25:07 on dist-test-slave-9zdj"
server_key: "1796af53025531cf53c8d01d16c7d13c"
server_key_iv: "0e99de91c171b3a84df3044f818ee6b5"
server_key_version: "encryptionkey@0"
I20260810 15:25:07.196187 29589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:25:07.197001 29604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:07.197225 29589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:07.197364 29589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e25d90a07f814a7a8159077ce022a01c"
format_stamp: "Formatted at 2026-08-10 15:25:07 on dist-test-slave-9zdj"
server_key: "1796af53025531cf53c8d01d16c7d13c"
server_key_iv: "0e99de91c171b3a84df3044f818ee6b5"
server_key_version: "encryptionkey@0"
I20260810 15:25:07.197448 29589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:07.211560 29589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:07.213510 29589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:07.213757 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:07.218088 29589 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:44655
I20260810 15:25:07.218138 29663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:44655 every 8 connection(s)
I20260810 15:25:07.218504 29589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:07.219079 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29589
I20260810 15:25:07.219195 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:07.219427 4353 external_mini_cluster.cc:1428] Setting key 3dbc8579287f1be579e2fa373cedfb16
I20260810 15:25:07.219547 29664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:07.223158 29664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c: Bootstrap starting.
I20260810 15:25:07.223807 29664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:07.224165 29664 log.cc:824] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:07.224944 29664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c: No bootstrap required, opened a new log
Aug 10 15:25:07 dist-test-slave-9zdj krb5kdc[29573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:25:07.226989 29664 raft_consensus.cc:348] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25d90a07f814a7a8159077ce022a01c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44655 } }
I20260810 15:25:07.227156 29664 raft_consensus.cc:374] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:07.227200 29664 raft_consensus.cc:729] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25d90a07f814a7a8159077ce022a01c, State: Initialized, Role: FOLLOWER
I20260810 15:25:07.227376 29664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25d90a07f814a7a8159077ce022a01c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44655 } }
I20260810 15:25:07.227509 29664 raft_consensus.cc:388] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:07.227591 29664 raft_consensus.cc:482] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:07.227654 29664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:07.228569 29664 raft_consensus.cc:504] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25d90a07f814a7a8159077ce022a01c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44655 } }
I20260810 15:25:07.228688 29664 leader_election.cc:302] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e25d90a07f814a7a8159077ce022a01c; no voters:
I20260810 15:25:07.228869 29664 leader_election.cc:288] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:07.228925 29671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:07.229128 29671 raft_consensus.cc:686] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [term 1 LEADER]: Becoming Leader. State: Replica: e25d90a07f814a7a8159077ce022a01c, State: Running, Role: LEADER
I20260810 15:25:07.229142 29664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:07.229300 29671 consensus_queue.cc:237] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25d90a07f814a7a8159077ce022a01c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44655 } }
I20260810 15:25:07.229668 29671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e25d90a07f814a7a8159077ce022a01c. Latest consensus state: current_term: 1 leader_uuid: "e25d90a07f814a7a8159077ce022a01c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25d90a07f814a7a8159077ce022a01c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44655 } } }
I20260810 15:25:07.229710 29672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e25d90a07f814a7a8159077ce022a01c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e25d90a07f814a7a8159077ce022a01c" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 44655 } } }
I20260810 15:25:07.229828 29671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:07.229836 29672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:07.230099 29680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:07.230674 29680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:07.231758 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:07.219984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47985 (local address 127.4.64.126:44655)
0810 15:25:07.220212 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:07.220223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:07.220258 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:25:07.220828 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:07.220838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:07.220911 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:07.221073 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:07.221900 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:07.223275 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:07.224096 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:07.224664 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:07.224781 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:07.227199 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:07.227203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:07.227203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:07.227216 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:07.229399 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:07.229921 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:07.229925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:07.229925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:07.230054 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:07.230290 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:07.230290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:07.230291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:07.230651 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:07.231316 (+ 665us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:07.231395 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 15:25:07.231547 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
I20260810 15:25:07.232298 29680 catalog_manager.cc:1347] Generated new cluster ID: bf0f3aa53f5848cfaa025f09d0707c6a
I20260810 15:25:07.232352 29680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:07.232939 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:07.238982 29680 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:07.239511 29680 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:07.242282 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:07.234532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49597 (local address 127.4.64.126:44655)
0810 15:25:07.234566 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:07.234568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:07.234614 (+ 46us) server_negotiation.cc:407] Connection header received
0810 15:25:07.234635 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:07.234635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:07.234646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:07.234697 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:07.235317 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:07.236093 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:07.237121 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:07.237474 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:07.237554 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:07.239199 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:07.239200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:07.239201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:07.239208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:07.240999 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:07.241390 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:07.241392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:07.241392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:07.241462 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:07.241718 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:07.241719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:07.241719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:07.241878 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:07.241982 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:07.242091 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 15:25:07.242186 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:25:07.246853 29680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e25d90a07f814a7a8159077ce022a01c: Generated new TSK 0
I20260810 15:25:07.247045 29680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:07.250474 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29589
2026-08-10T15:25:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:07 dist-test-slave-9zdj krb5kdc[29702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:07 dist-test-slave-9zdj krb5kdc[29702](info): set up 2 sockets
Aug 10 15:25:07 dist-test-slave-9zdj krb5kdc[29702](info): commencing operation
krb5kdc: starting...
W20260810 15:25:08.378949 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:08 dist-test-slave-9zdj krb5kdc[29702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:08.513526 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43671
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:08.599208 29717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:08.599434 29717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:08.599457 29717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:08.600625 29717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:08.600672 29717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:08.600687 29717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:08.600699 29717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:08.600713 29717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:08.602267 29717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:08.602536 29717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:08.602790 29717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:08.608358 29722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:08.608426 29717 server_base.cc:1034] running on GCE node
W20260810 15:25:08.608397 29723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:08.608366 29725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:08.608879 29717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:08.609169 29717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:08.610364 29717 hybrid_clock.cc:648] HybridClock initialized: now 1786375508610332 us; error 44 us; skew 500 ppm
Aug 10 15:25:08 dist-test-slave-9zdj krb5kdc[29702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375508, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:08.616993 29717 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:08.617551 29717 webserver.cc:460] Webserver started at http://127.4.64.126:34923/ using document root <none> and password file <none>
I20260810 15:25:08.617846 29717 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:08.617924 29717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:08.618163 29717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:08.619344 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "6a09be84149a453aa393b12b4359aa11"
format_stamp: "Formatted at 2026-08-10 15:25:08 on dist-test-slave-9zdj"
server_key: "3fd51ea875760692d9fd171544ece797"
server_key_iv: "c88cba67ba08c8d0d9e9ef4062981f43"
server_key_version: "encryptionkey@0"
I20260810 15:25:08.619742 29717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "6a09be84149a453aa393b12b4359aa11"
format_stamp: "Formatted at 2026-08-10 15:25:08 on dist-test-slave-9zdj"
server_key: "3fd51ea875760692d9fd171544ece797"
server_key_iv: "c88cba67ba08c8d0d9e9ef4062981f43"
server_key_version: "encryptionkey@0"
I20260810 15:25:08.621234 29717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:25:08.622145 29733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:08.622416 29717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:25:08.622535 29717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "6a09be84149a453aa393b12b4359aa11"
format_stamp: "Formatted at 2026-08-10 15:25:08 on dist-test-slave-9zdj"
server_key: "3fd51ea875760692d9fd171544ece797"
server_key_iv: "c88cba67ba08c8d0d9e9ef4062981f43"
server_key_version: "encryptionkey@0"
I20260810 15:25:08.622617 29717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:08.638734 29717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:08.641360 29717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:08.641525 29717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:08.645498 29717 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:43671
I20260810 15:25:08.645533 29792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:43671 every 8 connection(s)
I20260810 15:25:08.645967 29717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:08.646972 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:08.649335 29793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11: Bootstrap starting.
I20260810 15:25:08.649991 29793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:08.650194 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29717
I20260810 15:25:08.650285 29793 log.cc:824] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:08.650343 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:08.650560 4353 external_mini_cluster.cc:1428] Setting key 15ff34825f5c2cb8f3d73d3f6ec6cdbd
I20260810 15:25:08.651460 29793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11: No bootstrap required, opened a new log
I20260810 15:25:08.654383 29793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09be84149a453aa393b12b4359aa11" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43671 } }
I20260810 15:25:08.654531 29793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:08.654554 29793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a09be84149a453aa393b12b4359aa11, State: Initialized, Role: FOLLOWER
I20260810 15:25:08.654690 29793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09be84149a453aa393b12b4359aa11" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43671 } }
I20260810 15:25:08.654772 29793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:08.654815 29793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:08.654865 29793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:08.655612 29793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09be84149a453aa393b12b4359aa11" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43671 } }
I20260810 15:25:08.655727 29793 leader_election.cc:302] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a09be84149a453aa393b12b4359aa11; no voters:
I20260810 15:25:08.655912 29793 leader_election.cc:288] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:08.655972 29800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:08.656198 29800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [term 1 LEADER]: Becoming Leader. State: Replica: 6a09be84149a453aa393b12b4359aa11, State: Running, Role: LEADER
I20260810 15:25:08.656265 29793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:08.656719 29800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09be84149a453aa393b12b4359aa11" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43671 } }
I20260810 15:25:08.657328 29801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a09be84149a453aa393b12b4359aa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09be84149a453aa393b12b4359aa11" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43671 } } }
I20260810 15:25:08.657442 29801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:08.657708 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:08.651158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55341 (local address 127.4.64.126:43671)
0810 15:25:08.651459 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:08.651468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:08.651503 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:25:08.652047 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:08.652058 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:08.652125 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:08.652288 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:08.652560 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:08.654096 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:08.655183 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:08.655852 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:08.655974 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:08.656761 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:08.656765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:08.656765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:08.657031 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:08.657074 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:08.657091 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:08.657155 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:08.657382 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 15:25:08.657455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":71,"threads_started":1}
I20260810 15:25:08.657753 29810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:08.658130 29802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a09be84149a453aa393b12b4359aa11. Latest consensus state: current_term: 1 leader_uuid: "6a09be84149a453aa393b12b4359aa11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09be84149a453aa393b12b4359aa11" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43671 } } }
I20260810 15:25:08.658224 29802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:08.658576 29810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:08.658818 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:08.660573 29810 catalog_manager.cc:1347] Generated new cluster ID: b8c298bba9044f1bb9f3ddb4e1507423
I20260810 15:25:08.660625 29810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:08.669467 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:08.662062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60675 (local address 127.4.64.126:43671)
0810 15:25:08.662098 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:08.662100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:08.665928 (+ 3828us) server_negotiation.cc:407] Connection header received
0810 15:25:08.665995 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:08.665996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:08.666008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:08.666072 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:08.666253 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:08.666989 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:08.668064 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:08.668368 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:08.668428 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:08.669194 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:08.669195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:08.669195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:08.669200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:08.669215 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:08.669219 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:08.669268 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:08.669358 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:25:08.669381 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:25:08.671985 29810 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:08.672533 29810 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:08.679836 29810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a09be84149a453aa393b12b4359aa11: Generated new TSK 0
I20260810 15:25:08.680042 29810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:08.685755 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29717
2026-08-10T15:25:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:08 dist-test-slave-9zdj krb5kdc[29829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:08 dist-test-slave-9zdj krb5kdc[29829](info): set up 2 sockets
Aug 10 15:25:08 dist-test-slave-9zdj krb5kdc[29829](info): commencing operation
krb5kdc: starting...
W20260810 15:25:09.834064 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:09 dist-test-slave-9zdj krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:09.977411 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35643
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:10.064630 29844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:10.064884 29844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:10.064909 29844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:10.066139 29844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:10.066193 29844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:10.066254 29844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:10.066269 29844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:10.066282 29844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:10.067836 29844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:35643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:10.068061 29844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:10.068259 29844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:25:10.073729 29844 server_base.cc:1034] running on GCE node
W20260810 15:25:10.073700 29849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:10.073697 29850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:10.073698 29852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:10.074301 29844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:10.074673 29844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:10.075870 29844 hybrid_clock.cc:648] HybridClock initialized: now 1786375510075847 us; error 39 us; skew 500 ppm
Aug 10 15:25:10 dist-test-slave-9zdj krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375510, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:10.078781 29844 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:10.079245 29844 webserver.cc:460] Webserver started at http://127.4.64.126:34443/ using document root <none> and password file <none>
I20260810 15:25:10.079443 29844 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:10.079486 29844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:10.079586 29844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:10.080681 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "8dc492f2fe6a4d08bbebda4e642d5af2"
format_stamp: "Formatted at 2026-08-10 15:25:10 on dist-test-slave-9zdj"
server_key: "4e2147568da48988bc24480e603c8533"
server_key_iv: "4aa09d1937e1c9a3830b3b5174607634"
server_key_version: "encryptionkey@0"
I20260810 15:25:10.081004 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "8dc492f2fe6a4d08bbebda4e642d5af2"
format_stamp: "Formatted at 2026-08-10 15:25:10 on dist-test-slave-9zdj"
server_key: "4e2147568da48988bc24480e603c8533"
server_key_iv: "4aa09d1937e1c9a3830b3b5174607634"
server_key_version: "encryptionkey@0"
I20260810 15:25:10.082722 29844 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 15:25:10.083501 29860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:10.083714 29844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:25:10.083812 29844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "8dc492f2fe6a4d08bbebda4e642d5af2"
format_stamp: "Formatted at 2026-08-10 15:25:10 on dist-test-slave-9zdj"
server_key: "4e2147568da48988bc24480e603c8533"
server_key_iv: "4aa09d1937e1c9a3830b3b5174607634"
server_key_version: "encryptionkey@0"
I20260810 15:25:10.083874 29844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:10.087330 29844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:10.089725 29844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:10.089855 29844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:10.093796 29844 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:35643
I20260810 15:25:10.093875 29918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:35643 every 8 connection(s)
I20260810 15:25:10.094236 29844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:10.095430 29919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:10.097883 29919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2: Bootstrap starting.
I20260810 15:25:10.098528 29919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:10.098834 29919 log.cc:824] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:10.099771 29919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2: No bootstrap required, opened a new log
I20260810 15:25:10.102358 29919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35643 } }
I20260810 15:25:10.102517 29919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:25:10.102540 29919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc492f2fe6a4d08bbebda4e642d5af2, State: Initialized, Role: FOLLOWER
I20260810 15:25:10.102711 29919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35643 } }
I20260810 15:25:10.102780 29919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:25:10.102803 29919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:25:10.102873 29919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:10.103802 29919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35643 } }
I20260810 15:25:10.103937 29919 leader_election.cc:302] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dc492f2fe6a4d08bbebda4e642d5af2; no voters:
I20260810 15:25:10.104017 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29844
I20260810 15:25:10.104173 29919 leader_election.cc:288] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:25:10.104180 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:10.104198 29922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:10.104406 29922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [term 1 LEADER]: Becoming Leader. State: Replica: 8dc492f2fe6a4d08bbebda4e642d5af2, State: Running, Role: LEADER
I20260810 15:25:10.104415 4353 external_mini_cluster.cc:1428] Setting key 640b6d7ca78ea3a2960e62244a16af19
I20260810 15:25:10.104528 29919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:10.104566 29922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35643 } }
I20260810 15:25:10.105516 29924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc492f2fe6a4d08bbebda4e642d5af2. Latest consensus state: current_term: 1 leader_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35643 } } }
I20260810 15:25:10.105599 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc492f2fe6a4d08bbebda4e642d5af2" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 35643 } } }
I20260810 15:25:10.105643 29924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:10.105741 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2 [sys.catalog]: This master's current role is: LEADER
W20260810 15:25:10.107402 29943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:25:10.107519 29943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:25:10.107649 29938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:10.107919 29938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:10.108180 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:10.105213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49273 (local address 127.4.64.126:35643)
0810 15:25:10.106429 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:10.106442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:10.106487 (+ 45us) server_negotiation.cc:407] Connection header received
0810 15:25:10.106622 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:10.106629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:10.106663 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:10.106826 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:10.107533 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:10.107536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:10.107536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:10.107541 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:10.107597 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:10.107628 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:10.107698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:10.107954 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 15:25:10.108002 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":68,"threads_started":1}
I20260810 15:25:10.109244 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:10.109827 29938 catalog_manager.cc:1347] Generated new cluster ID: 2c1eadaaa9334e3693fa084664fde731
I20260810 15:25:10.109892 29938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:10.115501 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:10.114083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57581 (local address 127.4.64.126:35643)
0810 15:25:10.114134 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:10.114136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:10.114363 (+ 227us) server_negotiation.cc:407] Connection header received
0810 15:25:10.114444 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:10.114445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:10.114450 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:10.114498 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:10.115198 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:10.115200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:10.115200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:25:10.115205 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:10.115221 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:25:10.115233 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:10.115308 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:10.115369 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 15:25:10.115385 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:25:10.122972 29938 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:10.123191 29938 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:10.130342 29938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dc492f2fe6a4d08bbebda4e642d5af2: Generated new TSK 0
I20260810 15:25:10.130589 29938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:10.134711 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29844
2026-08-10T15:25:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1448 ms)
[----------] 18 tests from AuthTokenIssuingTest (26490 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:10 dist-test-slave-9zdj krb5kdc[29954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:10 dist-test-slave-9zdj krb5kdc[29954](info): set up 2 sockets
Aug 10 15:25:10 dist-test-slave-9zdj krb5kdc[29954](info): commencing operation
krb5kdc: starting...
W20260810 15:25:11.262442 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:11.396994 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43831
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43831,127.4.64.125:32859,127.4.64.124:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:11.482556 29969 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:25:11.482847 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:11.482882 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:11.482926 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:11.484071 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:11.484162 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:11.484190 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:11.484222 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:11.484244 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:11.485807 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43831,127.4.64.125:32859,127.4.64.124:33497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:11.486107 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:11.486366 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:11.491569 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:11.491559 29978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:11.491559 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:11.491941 29969 server_base.cc:1034] running on GCE node
I20260810 15:25:11.492236 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:11.492571 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:11.493743 29969 hybrid_clock.cc:648] HybridClock initialized: now 1786375511493729 us; error 34 us; skew 500 ppm
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:11.496421 29969 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:11.496907 29969 webserver.cc:460] Webserver started at http://127.4.64.126:33527/ using document root <none> and password file <none>
I20260810 15:25:11.497103 29969 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:11.497144 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:11.497238 29969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:11.498375 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "e3838eb54e5e488899fd371e590d82b1"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "207859674e15c14ae5f82042d4dfb58f"
server_key_iv: "0820021fcacdfe6d058312a4c7de265b"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.498694 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "e3838eb54e5e488899fd371e590d82b1"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "207859674e15c14ae5f82042d4dfb58f"
server_key_iv: "0820021fcacdfe6d058312a4c7de265b"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.500022 29969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:25:11.500751 29986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:11.501010 29969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:11.501117 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "e3838eb54e5e488899fd371e590d82b1"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "207859674e15c14ae5f82042d4dfb58f"
server_key_iv: "0820021fcacdfe6d058312a4c7de265b"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.501206 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:11.512835 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:11.515218 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:11.515401 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:11.519620 29969 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:43831
I20260810 15:25:11.519658 30048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:43831 every 8 connection(s)
I20260810 15:25:11.520072 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:11.521284 30049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:11.522966 30049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.523293 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 29969
I20260810 15:25:11.523440 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:11.523658 4353 external_mini_cluster.cc:1428] Setting key 0a52734d643feb60cfd20a68fef59fa5
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.125@KRBTEST.COM; defaulting to no policy
I20260810 15:25:11.534574 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.523836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:43831 (local address 127.0.0.1:32846)
0810 15:25:11.524031 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.524069 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.524671 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.524974 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.524977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.525300 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.525557 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.525563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.526655 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.526656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.527418 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.527421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.527950 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.527951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.528016 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.528697 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.528708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.530557 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.532675 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.532683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.532686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.532922 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.533208 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.533209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.533209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.533305 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.533796 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.533802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.533876 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.533999 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.534363 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":195,"thread_start_us":108,"threads_started":1}
I20260810 15:25:11.534578 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.524066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.4.64.126:43831)
0810 15:25:11.524314 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.524319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.524359 (+ 40us) server_negotiation.cc:407] Connection header received
0810 15:25:11.524838 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.524840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.524874 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.524969 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.525707 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.526562 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.527520 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.527844 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.527964 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.530650 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.530658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.530658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.530665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.532517 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.533019 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.533023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.533024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.533120 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.533379 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.533380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.533380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.533719 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.533822 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.534041 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.534289 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.125@KRBTEST.COM" created.
I20260810 15:25:11.535593 30049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.536056 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.535897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60588)
0810 15:25:11.535958 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.536022 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 15:25:11.536237 29991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
I20260810 15:25:11.536783 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.536657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60592)
0810 15:25:11.536710 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.536747 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 15:25:11.536916 30049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:32859: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab.
Entry for principal kudu/127.4.64.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab.
I20260810 15:25:11.553835 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:32859
--webserver_interface=127.4.64.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab
--principal=kudu/127.4.64.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43831,127.4.64.125:32859,127.4.64.124:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:25:11.586156 30049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } attempt: 1
I20260810 15:25:11.586737 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.586583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60596)
0810 15:25:11.586655 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.586705 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 15:25:11.587154 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.587057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60600)
0810 15:25:11.587105 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.587129 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 15:25:11.587327 30049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:32859: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
W20260810 15:25:11.641444 30058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:25:11.641695 30058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:11.641716 30058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:11.641736 30058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:11.642807 30058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:11.642841 30058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:11.642855 30058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:11.642868 30058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:11.642881 30058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:11.644347 30058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43831,127.4.64.125:32859,127.4.64.124:33497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab
--principal=kudu/127.4.64.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.64.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:11.644553 30058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:11.644737 30058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:11.651060 30063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:11.651218 30067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:11.651060 30069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:11.651280 30058 server_base.cc:1034] running on GCE node
I20260810 15:25:11.651731 30058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:11.652009 30058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:11.653178 30058 hybrid_clock.cc:648] HybridClock initialized: now 1786375511653159 us; error 40 us; skew 500 ppm
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:11.655622 30049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } attempt: 2
I20260810 15:25:11.655946 30058 init.cc:377] Logged in from keytab as kudu/127.4.64.125@KRBTEST.COM (short username kudu)
I20260810 15:25:11.656106 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.655944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60604)
0810 15:25:11.656013 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.656070 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:25:11.656507 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.656371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60612)
0810 15:25:11.656421 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.656476 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:25:11.656535 30058 webserver.cc:460] Webserver started at http://127.4.64.125:43627/ using document root <none> and password file <none>
W20260810 15:25:11.656615 30049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:32859: Network error: Client connection negotiation failed: client connection to 127.4.64.125:32859: connect: Connection refused (error 111)
I20260810 15:25:11.656812 30058 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:11.656891 30058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:11.657038 30058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:11.658254 30058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data/instance:
uuid: "31060f871056483cb8fdf5a929a7bdb5"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "a046fb21644b1a3f69106348b332744c"
server_key_iv: "25f654aa6dc8d8e76391f61a9d1e2846"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.658640 30058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal/instance:
uuid: "31060f871056483cb8fdf5a929a7bdb5"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "a046fb21644b1a3f69106348b332744c"
server_key_iv: "25f654aa6dc8d8e76391f61a9d1e2846"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.660089 30058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:25:11.660897 30076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:11.661082 30058 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:11.661180 30058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal
uuid: "31060f871056483cb8fdf5a929a7bdb5"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "a046fb21644b1a3f69106348b332744c"
server_key_iv: "25f654aa6dc8d8e76391f61a9d1e2846"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.661260 30058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:11.674758 30058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:11.676822 30058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:11.677003 30058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:11.681496 30058 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.125:32859
I20260810 15:25:11.681550 30135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.125:32859 every 8 connection(s)
I20260810 15:25:11.681942 30058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
I20260810 15:25:11.683049 30136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:11.684636 30136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.690595 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 30058
I20260810 15:25:11.690722 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-1/wal/instance
I20260810 15:25:11.690963 4353 external_mini_cluster.cc:1428] Setting key 8a6cd10b4e613015433a496299185e66
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:25:11.695346 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.685539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.4.64.126:43831)
0810 15:25:11.685668 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.685670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.685860 (+ 190us) server_negotiation.cc:407] Connection header received
0810 15:25:11.686445 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.686446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.686460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.686512 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.687198 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.687884 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.688910 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.689136 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.689218 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.691941 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.691943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.691943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.691953 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.693502 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.694078 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.694079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.694080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.694234 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.694590 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.694591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.694592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.694720 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.694780 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.695186 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.695269 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 15:25:11.695606 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.685424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:43831 (local address 127.0.0.1:32862)
0810 15:25:11.685716 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.685765 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.686319 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.686592 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.686596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.686871 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.687082 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.687086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.687986 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.687987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.688813 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.688816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.689247 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.689249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.689371 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.690010 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.690019 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.691831 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.693592 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.693600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.693602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.693990 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.694319 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.694320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.694320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.694485 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.694791 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.694798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.694988 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.695104 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.695302 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260810 15:25:11.696257 30136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.64.124@KRBTEST.COM; defaulting to no policy
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.124@KRBTEST.COM" created.
I20260810 15:25:11.704562 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.696602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.4.64.125:32859)
0810 15:25:11.696787 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.696792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.696811 (+ 19us) server_negotiation.cc:407] Connection header received
0810 15:25:11.696836 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.696837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.696849 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.696916 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.697282 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.698044 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.699065 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.699463 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.699538 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.701713 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.701721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.701721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.701728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.703260 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.703602 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.703605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.703606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.703678 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.703853 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.703853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.703854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.704069 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.704256 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.704391 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.704467 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260810 15:25:11.704571 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.696564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60620)
0810 15:25:11.696617 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.696631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.696684 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.696941 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.696942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.697126 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.697202 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.697204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.698127 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.698128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.698971 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.698974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.699535 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.699537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.699651 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.700149 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.700162 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.701590 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.703347 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.703348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.703348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.703520 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.703745 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.703746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.703746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.703800 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.704168 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.704169 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.704223 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.704294 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.704404 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:25:11.705425 30136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.705837 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.705693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34014)
0810 15:25:11.705748 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.705804 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:25:11.705999 30078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
I20260810 15:25:11.706480 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.706363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34026)
0810 15:25:11.706416 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.706450 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 15:25:11.706591 30136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:33497: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab.
Entry for principal kudu/127.4.64.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab.
I20260810 15:25:11.721520 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:33497
--webserver_interface=127.4.64.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab
--principal=kudu/127.4.64.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43831,127.4.64.125:32859,127.4.64.124:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:25:11.747797 30049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } attempt: 3
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:25:11.755638 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.748314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.4.64.125:32859)
0810 15:25:11.748386 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.748388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.748400 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:25:11.748480 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.748481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.748493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.748567 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.748913 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.749427 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.750231 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.750482 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.750564 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.752680 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.752681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.752682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.752688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.754147 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.754509 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.754510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.754510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.754572 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.755019 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.755021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.755021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.755160 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.755226 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.755451 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.755542 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:25:11.755664 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.748238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60630)
0810 15:25:11.748310 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.748321 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.748405 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.748575 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.748576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.748753 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.748820 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.748822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.749517 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.749518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.750128 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.750130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.750562 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.750563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.750645 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.751252 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.751257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.752559 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.754239 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.754240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.754241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.754421 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.754666 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.754667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.754668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.754877 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.755282 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.755283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.755337 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.755423 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.755538 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 15:25:11.755832 30136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } attempt: 1
I20260810 15:25:11.756232 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.756105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34034)
0810 15:25:11.756165 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.756204 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 15:25:11.756482 30049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.756891 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.756742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34040)
0810 15:25:11.756799 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.756834 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:25:11.756951 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.756846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34052)
0810 15:25:11.756896 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.756925 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 15:25:11.757027 30136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:33497: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
I20260810 15:25:11.757246 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.757150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34062)
0810 15:25:11.757186 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.757216 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 15:25:11.757345 30049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:33497: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
I20260810 15:25:11.788530 30049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } attempt: 1
I20260810 15:25:11.789126 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.788945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34066)
0810 15:25:11.789021 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.789079 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 15:25:11.789669 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.789524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34076)
0810 15:25:11.789572 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.789606 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 15:25:11.789789 30049 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:33497: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
W20260810 15:25:11.810036 30144 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:25:11.810279 30144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:11.810298 30144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:11.810321 30144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:11.811543 30144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:11.811589 30144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:11.811602 30144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:11.811616 30144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:11.811628 30144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:11.813274 30144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43831,127.4.64.125:32859,127.4.64.124:33497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab
--principal=kudu/127.4.64.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.64.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:11.813568 30144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:11.813843 30144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:11.819015 30150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:11.819041 30152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:11.819041 30155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:11.819360 30144 server_base.cc:1034] running on GCE node
I20260810 15:25:11.819655 30144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:11.819964 30144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:11.821137 30144 hybrid_clock.cc:648] HybridClock initialized: now 1786375511821107 us; error 67 us; skew 500 ppm
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:11.823850 30144 init.cc:377] Logged in from keytab as kudu/127.4.64.124@KRBTEST.COM (short username kudu)
I20260810 15:25:11.824361 30144 webserver.cc:460] Webserver started at http://127.4.64.124:43987/ using document root <none> and password file <none>
I20260810 15:25:11.824611 30144 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:11.824679 30144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:11.824827 30144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:11.825158 30136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } attempt: 2
I20260810 15:25:11.825716 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.825538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34092)
0810 15:25:11.825601 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.825655 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:25:11.826040 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.825945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34106)
0810 15:25:11.825990 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.826015 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 15:25:11.826133 30136 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:33497: Network error: Client connection negotiation failed: client connection to 127.4.64.124:33497: connect: Connection refused (error 111)
I20260810 15:25:11.826167 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data/instance:
uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "7a0ca41da333fa7dc8d2a3f4283bd370"
server_key_iv: "bf40560f7d35587a1f3fb750b0b9cedd"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.826576 30144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal/instance:
uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "7a0ca41da333fa7dc8d2a3f4283bd370"
server_key_iv: "bf40560f7d35587a1f3fb750b0b9cedd"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.828109 30144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:25:11.829084 30166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:11.829356 30144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:11.829476 30144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal
uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da"
format_stamp: "Formatted at 2026-08-10 15:25:11 on dist-test-slave-9zdj"
server_key: "7a0ca41da333fa7dc8d2a3f4283bd370"
server_key_iv: "bf40560f7d35587a1f3fb750b0b9cedd"
server_key_version: "encryptionkey@0"
I20260810 15:25:11.829562 30144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:11.846871 30144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:11.849552 30144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:11.849773 30144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:11.854460 30144 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.124:33497
I20260810 15:25:11.854590 30225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.124:33497 every 8 connection(s)
I20260810 15:25:11.854967 30144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
I20260810 15:25:11.856045 30226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:11.857746 30226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.858239 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 30144
I20260810 15:25:11.858330 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786375228510355-4353-0/minicluster-data/master-2/wal/instance
I20260810 15:25:11.858597 4353 external_mini_cluster.cc:1428] Setting key 50268e378919d057e2f889de0211f95a
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:25:11.865018 30049 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } attempt: 2
I20260810 15:25:11.868479 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.858551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.4.64.126:43831)
0810 15:25:11.858656 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.858659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.858919 (+ 260us) server_negotiation.cc:407] Connection header received
0810 15:25:11.859532 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.859533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.859546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.859600 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.860162 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.860655 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.861549 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.861844 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.861950 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.864454 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.864455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.864455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.864461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.866300 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.866840 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.866841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.866841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.866908 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.867307 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.867308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.867309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.867456 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.867527 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.868303 (+ 776us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.868404 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:25:11.868707 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.858551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:43831 (local address 127.0.0.1:32864)
0810 15:25:11.858774 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.858817 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.859418 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.859618 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.859620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.859900 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.860064 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.860069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.860738 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.860739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.861455 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.861457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.861980 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.861981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.862100 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.862634 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.862645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.864370 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.866374 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.866382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.866384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.866732 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.867067 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.867068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.867068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.867217 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.867972 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.867978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.868147 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.868263 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.868441 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":138,"threads_started":1}
I20260810 15:25:11.869364 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.858996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.4.64.124:33497)
0810 15:25:11.859161 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.859167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.859277 (+ 110us) server_negotiation.cc:407] Connection header received
0810 15:25:11.859428 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.859434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.859453 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.859543 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.860209 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.860995 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.861829 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.862149 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.862249 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.865147 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.865155 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.865155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.865162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.867690 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.868087 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.868090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.868090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.868164 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.868393 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.868394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.868395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.868558 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.868690 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.869173 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.869263 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260810 15:25:11.869477 30226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:25:11.874120 30052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.865299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34116)
0810 15:25:11.865344 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.865355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.865414 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.865771 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.865772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.865940 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.865984 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.865985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.866773 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.866774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.867443 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.867445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.867960 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.867961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.868024 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.868636 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.868641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.870264 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.872673 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.872674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.872674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.872911 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.873143 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.873144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.873144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.873206 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.873488 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.873488 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.873816 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.873907 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.874012 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:25:11.874191 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.865433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.4.64.124:33497)
0810 15:25:11.865595 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.865597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.865636 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:25:11.865657 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.865658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.865668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.866054 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.866096 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.866689 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.867531 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.867879 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.867931 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.870346 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.870347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.870348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.870354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.872587 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.872993 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.872994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.872995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.873063 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.873276 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.873276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.873277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.873399 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.873500 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.873963 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.874047 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:25:11.875792 30049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1: Bootstrap starting.
I20260810 15:25:11.876470 30049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:11.876768 30049 log.cc:824] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:11.877547 30049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1: No bootstrap required, opened a new log
I20260810 15:25:11.877568 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.869981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.4.64.125:32859)
0810 15:25:11.870033 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.870036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.870045 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:25:11.870070 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.870071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.870081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.870143 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.870564 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.871324 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.872312 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.872672 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.872740 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.874936 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.874938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.874938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.874946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.876390 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.876776 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.876778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.876778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.876843 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.877054 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.877055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.877055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.877174 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.877247 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.877409 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.877482 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:25:11.877568 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.869790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:32859 (local address 127.0.0.1:60646)
0810 15:25:11.869842 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.869852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.869893 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.870262 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.870263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.870416 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.870474 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.870476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.871425 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.871426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.872218 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.872221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.872739 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.872739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.872814 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.873386 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.873390 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.874842 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.876497 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.876498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.876499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.876699 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.876912 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.876913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.876914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.876972 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.877249 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.877250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.877293 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.877382 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.877476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 15:25:11.877888 30226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:11.879650 30049 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.879766 30049 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3838eb54e5e488899fd371e590d82b1, State: Initialized, Role: FOLLOWER
I20260810 15:25:11.879823 30049 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.880095 30233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.880175 30049 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:11.880191 30233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:25:11.882056 30245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
W20260810 15:25:11.882169 30245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:25:11.885878 30228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.878162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34120)
0810 15:25:11.878212 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.878222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.878257 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.878387 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.878387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.878495 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.878523 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.878524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.879399 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.879400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.880008 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.880009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.880500 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.880500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.880548 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.881006 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.881011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.882495 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.884571 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.884572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.884572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.884809 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.885040 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.885041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.885041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.885108 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.885485 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.885486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.885533 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.885647 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.885759 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:25:11.885965 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.878240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.4.64.124:33497)
0810 15:25:11.878273 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.878275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.878283 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:25:11.878306 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.878307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.878315 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.878386 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.878591 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.879294 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.880108 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.880423 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.880504 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.882584 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.882585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.882586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.882592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.884469 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.884897 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.884898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.884899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.884963 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.885199 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.885201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.885201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.885405 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.885484 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.885696 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.885826 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":18}
I20260810 15:25:11.883461 29969 master_runner.cc:186] Error getting master registration for 127.4.64.125:32859: OK, instance_id { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" instance_seqno: 1786375511680123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:25:11.886452 29969 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:25:11.887513 30226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: Bootstrap starting.
I20260810 15:25:11.888301 30226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:11.888633 30226 log.cc:824] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:11.889505 30226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: No bootstrap required, opened a new log
I20260810 15:25:11.889808 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.870555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.4.64.126:43831)
0810 15:25:11.870825 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.870827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.870833 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:25:11.870849 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.870849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.870856 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.870959 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.871484 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.872164 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.872999 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.873272 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.873324 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.886931 (+ 13607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.886932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.886932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.886940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.888415 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.888773 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.888774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.888774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.888828 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.889065 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.889066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.889066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.889206 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.889300 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.889508 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.889591 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260810 15:25:11.891883 30226 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.892059 30226 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d34425c7f54a7f9bfc3cb5a466b6da, State: Initialized, Role: FOLLOWER
I20260810 15:25:11.892215 30226 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.892622 30226 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:11.892582 30249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.892792 30249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
W20260810 15:25:11.895220 30261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:25:11.895310 30261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:25:11.899238 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.890404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.4.64.125:32859)
0810 15:25:11.890447 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.890449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.890486 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:25:11.890559 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.890560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.890569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.890613 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.891101 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.891575 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.892331 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.892594 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.892662 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.895667 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.895669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.895670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.895678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.897570 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.898055 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.898056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.898057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.898117 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.898310 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.898311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.898311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.898454 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.898614 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.899074 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.899153 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:25:11.896847 30144 master_runner.cc:186] Error getting master registration for 127.4.64.125:32859: OK, instance_id { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" instance_seqno: 1786375511680123 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:25:11.899574 30144 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:25:11.900697 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:11.901556 30021 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e3838eb54e5e488899fd371e590d82b1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32876
I20260810 15:25:11.901733 30021 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 15:25:11.901793 30021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:11.902530 30021 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.902768 30021 leader_election.cc:288] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [CANDIDATE]: Term 1 election: Requested vote from peers 31060f871056483cb8fdf5a929a7bdb5 (127.4.64.125:32859), c8d34425c7f54a7f9bfc3cb5a466b6da (127.4.64.124:33497)
I20260810 15:25:11.903273 30197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3838eb54e5e488899fd371e590d82b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da"
I20260810 15:25:11.903388 30197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:11.904395 30197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3838eb54e5e488899fd371e590d82b1 in term 1.
I20260810 15:25:11.904712 29988 leader_election.cc:302] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8d34425c7f54a7f9bfc3cb5a466b6da, e3838eb54e5e488899fd371e590d82b1; no voters:
I20260810 15:25:11.904881 30233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:11.905046 30233 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [term 1 LEADER]: Becoming Leader. State: Replica: e3838eb54e5e488899fd371e590d82b1, State: Running, Role: LEADER
I20260810 15:25:11.905170 30233 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.906030 30268 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3838eb54e5e488899fd371e590d82b1. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.906337 30268 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:11.908996 30271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:11.909266 30271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:11.917265 30136 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } attempt: 3
I20260810 15:25:11.918942 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.906725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.4.64.124:33497)
0810 15:25:11.906766 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.906768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.909751 (+ 2983us) server_negotiation.cc:407] Connection header received
0810 15:25:11.909850 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.909851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.909861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.909890 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.911186 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.911990 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.913001 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.913303 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.913412 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.915260 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.915262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.915262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.915270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.917603 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.918053 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.918055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.918056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.918117 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.918319 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.918320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.918320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.918479 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.918608 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.918732 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.918833 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:25:11.921635 30197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [term 1 FOLLOWER]: Refusing update from remote peer e3838eb54e5e488899fd371e590d82b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:25:11.922087 30268 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:25:11.923642 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.904412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.4.64.125:32859)
0810 15:25:11.904464 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.904466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.906356 (+ 1890us) server_negotiation.cc:407] Connection header received
0810 15:25:11.906499 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.906501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.906526 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.906562 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.907906 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.908648 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.911419 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.912201 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.912244 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.916482 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.916485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.916485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.916495 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.921079 (+ 4584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.922502 (+ 1423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.922504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.922504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.922572 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.922891 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.922892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.922892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.923038 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.923125 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.923444 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.923530 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:25:11.923717 30249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3838eb54e5e488899fd371e590d82b1. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.923799 30249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 15:25:11 dist-test-slave-9zdj krb5kdc[29954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375511, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:25:11.929111 30053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.905782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.4.64.126:43831)
0810 15:25:11.905865 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.905867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.907032 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 15:25:11.907136 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.907137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.907148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.907179 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.907861 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.908628 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.912409 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.912722 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.912762 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.920068 (+ 7306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.920069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.920070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.920078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.921883 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.925415 (+ 3532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.925418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.925418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.925488 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.925794 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.925795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.925795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.925937 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.926026 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.926646 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.929001 (+ 2355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 15:25:11.930296 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.917596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:33497 (local address 127.0.0.1:34142)
0810 15:25:11.917765 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:11.917776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:11.917819 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:11.918064 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:11.918065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:11.918253 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:11.918301 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.918302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.919510 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.919511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.920351 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:11.920354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.921178 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.921179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:11.921244 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.924214 (+ 2970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:11.924219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:11.926358 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:11.928979 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.928980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.928980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.929209 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.929601 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:11.929602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:11.929603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:11.929703 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:11.929974 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:11.929975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:11.930026 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 15:25:11.930090 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:25:11.930197 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260810 15:25:11.930296 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:11.917893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.4.64.124:33497)
0810 15:25:11.917931 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.917933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.917940 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:25:11.917959 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.917960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.917969 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.918021 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.918385 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.919154 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.920451 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:11.920774 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:11.920846 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:11.926478 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:11.926480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:11.926480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:11.926488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:11.928869 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.929478 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.929479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.929479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.929541 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:11.929760 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:11.929761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:11.929761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:11.929897 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:11.929968 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:11.930128 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 15:25:11.930208 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:25:11.931350 30233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.931450 30233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:11.931535 30249 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.931639 30249 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:11.931777 30274 mvcc.cc:205] Tried to move back new op lower bound from 7316994096824414208 to 7316994096764342272. Current Snapshot: MvccSnapshot[applied={T|T < 7316994096824414208}]
I20260810 15:25:11.932634 30136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: Bootstrap starting.
I20260810 15:25:11.933399 30136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:11.933743 30136 log.cc:824] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:11.934245 30273 mvcc.cc:205] Tried to move back new op lower bound from 7316994096824414208 to 7316994096764342272. Current Snapshot: MvccSnapshot[applied={T|T < 7316994096824414208}]
I20260810 15:25:11.938814 30271 catalog_manager.cc:1347] Generated new cluster ID: b3bc83b82e9446c18abae4b33ee1f1ad
I20260810 15:25:11.938879 30271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:11.939131 30136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: No bootstrap required, opened a new log
I20260810 15:25:11.941064 30136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.941203 30136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31060f871056483cb8fdf5a929a7bdb5, State: Initialized, Role: FOLLOWER
I20260810 15:25:11.941272 30136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } }
I20260810 15:25:11.941731 30107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:11.942620 30107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [term 1 FOLLOWER]: Refusing update from remote peer e3838eb54e5e488899fd371e590d82b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:25:11.942955 30268 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:25:11.941601 30136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:11.945473 30108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e3838eb54e5e488899fd371e590d82b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31060f871056483cb8fdf5a929a7bdb5"
W20260810 15:25:11.947487 30295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:25:11.947600 30295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:25:11.948074 30271 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:11.948824 30271 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:11.953732 30279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.953985 30279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:11.954077 30279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3838eb54e5e488899fd371e590d82b1. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.954310 30279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:11.954380 30279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.954464 30279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:11.957684 30281 mvcc.cc:205] Tried to move back new op lower bound from 7316994096824414208 to 7316994096764342272. Current Snapshot: MvccSnapshot[applied={T|T < 7316994096824414208}]
I20260810 15:25:11.958408 30271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1: Generated new TSK 0
I20260810 15:25:11.958670 30271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:25:11.959311 30268 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3838eb54e5e488899fd371e590d82b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3838eb54e5e488899fd371e590d82b1" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43831 } } peers { permanent_uuid: "31060f871056483cb8fdf5a929a7bdb5" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 32859 } } peers { permanent_uuid: "c8d34425c7f54a7f9bfc3cb5a466b6da" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 33497 } } }
I20260810 15:25:11.959405 30268 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3838eb54e5e488899fd371e590d82b1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:11.959649 30108 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e3838eb54e5e488899fd371e590d82b1 for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260810 15:25:11.988091 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:11.986289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.4.64.125:32859)
0810 15:25:11.986345 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.986347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.987338 (+ 991us) server_negotiation.cc:407] Connection header received
0810 15:25:11.987449 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.987451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.987463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.987496 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.988015 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60690: BlockingRecv error: recv got EOF from 127.0.0.1:60690 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 15:25:11.989097 30053 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:11.986733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.4.64.126:43831)
0810 15:25:11.986777 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.986779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.988537 (+ 1758us) server_negotiation.cc:407] Connection header received
0810 15:25:11.988649 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.988650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.988663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.988693 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.989027 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32906: BlockingRecv error: recv got EOF from 127.0.0.1:32906 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:25:11.990032 30229 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:11.987081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.4.64.124:33497)
0810 15:25:11.987143 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.987145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.989471 (+ 2326us) server_negotiation.cc:407] Connection header received
0810 15:25:11.989544 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.989545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.989558 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.989673 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.989970 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34150: BlockingRecv error: recv got EOF from 127.0.0.1:34150 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 15:25:11.992681 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:11.991918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.4.64.125:32859)
0810 15:25:11.991958 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.991960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.992156 (+ 196us) server_negotiation.cc:407] Connection header received
0810 15:25:11.992264 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.992265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.992285 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.992313 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.992618 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60700: BlockingRecv error: recv got EOF from 127.0.0.1:60700 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:25:11.995810 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:11.995013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.4.64.125:32859)
0810 15:25:11.995048 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.995050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.995305 (+ 255us) server_negotiation.cc:407] Connection header received
0810 15:25:11.995395 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.995396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.995407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.995434 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.995764 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60706: BlockingRecv error: recv got EOF from 127.0.0.1:60706 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:25:11.999809 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:11.999000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.4.64.125:32859)
0810 15:25:11.999050 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:11.999052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:11.999303 (+ 251us) server_negotiation.cc:407] Connection header received
0810 15:25:11.999395 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:11.999396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:11.999409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:11.999436 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:11.999763 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60708: BlockingRecv error: recv got EOF from 127.0.0.1:60708 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:25:12.006249 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.005245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.4.64.125:32859)
0810 15:25:12.005288 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.005290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.005591 (+ 301us) server_negotiation.cc:407] Connection header received
0810 15:25:12.005754 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.005755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.005767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.005797 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.006187 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60714: BlockingRecv error: recv got EOF from 127.0.0.1:60714 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 15:25:12.016810 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.015772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.4.64.125:32859)
0810 15:25:12.015818 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.015821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.016145 (+ 324us) server_negotiation.cc:407] Connection header received
0810 15:25:12.016259 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.016260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.016274 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.016305 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.016750 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60716: BlockingRecv error: recv got EOF from 127.0.0.1:60716 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:25:12.035565 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.034530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.4.64.125:32859)
0810 15:25:12.034578 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.034580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.034926 (+ 346us) server_negotiation.cc:407] Connection header received
0810 15:25:12.035039 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.035040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.035053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.035085 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.035504 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60732: BlockingRecv error: recv got EOF from 127.0.0.1:60732 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 15:25:12.070170 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.069152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.4.64.125:32859)
0810 15:25:12.069209 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.069211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.069508 (+ 297us) server_negotiation.cc:407] Connection header received
0810 15:25:12.069606 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.069607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.069661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.069693 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.070103 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60748: BlockingRecv error: recv got EOF from 127.0.0.1:60748 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260810 15:25:12.136734 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.135829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.4.64.125:32859)
0810 15:25:12.135922 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.135925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.136079 (+ 154us) server_negotiation.cc:407] Connection header received
0810 15:25:12.136142 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.136142 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.136154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.136231 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.136633 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60760: BlockingRecv error: recv got EOF from 127.0.0.1:60760 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 15:25:12.267009 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.266110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.4.64.125:32859)
0810 15:25:12.266209 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.266211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.266395 (+ 184us) server_negotiation.cc:407] Connection header received
0810 15:25:12.266481 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.266482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.266495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.266551 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.266939 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60774: BlockingRecv error: recv got EOF from 127.0.0.1:60774 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 15:25:12.525169 30140 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:12.524336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60790 (local address 127.4.64.125:32859)
0810 15:25:12.524438 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:12.524440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:12.524468 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:25:12.524575 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:12.524576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:12.524587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:12.524665 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:12.525085 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60790: BlockingRecv error: recv got EOF from 127.0.0.1:60790 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 15:25:12.895848 30261 catalog_manager.cc:1259] Loaded cluster ID: b3bc83b82e9446c18abae4b33ee1f1ad
I20260810 15:25:12.895938 30261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: loading cluster ID for follower catalog manager: success
I20260810 15:25:12.896829 30261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: acquiring CA information for follower catalog manager: success
I20260810 15:25:12.897183 30261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8d34425c7f54a7f9bfc3cb5a466b6da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:25:12.948140 30295 catalog_manager.cc:1259] Loaded cluster ID: b3bc83b82e9446c18abae4b33ee1f1ad
I20260810 15:25:12.948228 30295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: loading cluster ID for follower catalog manager: success
I20260810 15:25:12.949122 30295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: acquiring CA information for follower catalog manager: success
I20260810 15:25:12.949476 30295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31060f871056483cb8fdf5a929a7bdb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:25:13.041864 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:13.038838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.4.64.125:32859)
0810 15:25:13.038983 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:13.038984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:13.038997 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:25:13.039044 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:13.039045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:13.039098 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:13.039200 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:13.039470 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.040005 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.040824 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.041025 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.041085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:13.041438 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:13.041567 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:13.041660 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:13.041771 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 15:25:13.041793 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260810 15:25:13.046851 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:13.044081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.4.64.124:33497)
0810 15:25:13.044188 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:13.044190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:13.044207 (+ 17us) server_negotiation.cc:407] Connection header received
0810 15:25:13.044270 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:13.044271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:13.044309 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:13.044387 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:13.044469 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.044954 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.045990 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.046212 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.046303 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:13.046476 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:13.046604 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:13.046705 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:13.046729 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 15:25:13.046752 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260810 15:25:13.052893 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:13.048812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.4.64.126:43831)
0810 15:25:13.049052 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:13.049054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:13.049158 (+ 104us) server_negotiation.cc:407] Connection header received
0810 15:25:13.049231 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:13.049232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:13.049299 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:13.049392 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:13.049586 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.050409 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.051702 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.052047 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.052112 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:13.052366 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:13.052494 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:13.052553 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:13.052756 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 15:25:13.052782 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":58,"threads_started":1}
I20260810 15:25:13.052956 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:13.048884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.4.64.125:32859)
0810 15:25:13.048945 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:13.048948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:13.049318 (+ 370us) server_negotiation.cc:407] Connection header received
0810 15:25:13.049449 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:13.049451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:13.049465 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:13.049528 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:13.049837 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.050643 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.052034 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.052365 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.052407 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:13.052625 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:13.052660 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:13.052707 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:13.052842 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:25:13.052873 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:25:13.053013 30389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:13.048998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.4.64.124:33497)
0810 15:25:13.049055 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:13.049058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:13.049305 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:25:13.049367 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:13.049368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:13.049380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:13.049435 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:13.049792 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.050624 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.052008 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:13.052339 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:13.052385 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:13.052625 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:13.052660 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:13.052693 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:13.052908 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 15:25:13.052930 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:25:13.055895 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 29969
I20260810 15:25:13.068368 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 30058
I20260810 15:25:13.076798 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 30144
2026-08-10T15:25:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2944 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:25:13 dist-test-slave-9zdj krb5kdc[30402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:25:13 dist-test-slave-9zdj krb5kdc[30402](info): set up 2 sockets
Aug 10 15:25:13 dist-test-slave-9zdj krb5kdc[30402](info): commencing operation
krb5kdc: starting...
W20260810 15:25:14.285144 4353 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/test-user.keytab.
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.64.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
Entry for principal kudu/127.4.64.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab.
I20260810 15:25:14.421074 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43947
--webserver_interface=127.4.64.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43947,127.4.64.125:43237,127.4.64.124:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:25:14.517071 30418 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:25:14.517367 30418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:14.517402 30418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:14.517448 30418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:14.518841 30418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:14.518925 30418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:14.518952 30418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:14.518975 30418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:14.519009 30418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:14.520747 30418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43947,127.4.64.125:43237,127.4.64.124:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.126.keytab
--principal=kudu/127.4.64.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.126:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.64.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:14.521108 30418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:14.521405 30418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:14.527261 30423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:14.527276 30424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:14.527275 30426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:14.527520 30418 server_base.cc:1034] running on GCE node
I20260810 15:25:14.527824 30418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:14.528199 30418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:14.529385 30418 hybrid_clock.cc:648] HybridClock initialized: now 1786375514529356 us; error 40 us; skew 500 ppm
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:14.543113 30418 init.cc:377] Logged in from keytab as kudu/127.4.64.126@KRBTEST.COM (short username kudu)
I20260810 15:25:14.543813 30418 webserver.cc:460] Webserver started at http://127.4.64.126:38093/ using document root <none> and password file <none>
I20260810 15:25:14.544116 30418 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:14.544207 30418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:14.544390 30418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:14.545746 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data/instance:
uuid: "37d542c072584a3381900ff0b200d96a"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "0efbdea1784152d03028654d4ab1118d"
server_key_iv: "04b773d36aa820cab7d808786fcc8785"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.546187 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal/instance:
uuid: "37d542c072584a3381900ff0b200d96a"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "0efbdea1784152d03028654d4ab1118d"
server_key_iv: "04b773d36aa820cab7d808786fcc8785"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.547844 30418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:25:14.548760 30434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:14.548979 30418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:14.549108 30418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal
uuid: "37d542c072584a3381900ff0b200d96a"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "0efbdea1784152d03028654d4ab1118d"
server_key_iv: "04b773d36aa820cab7d808786fcc8785"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.549283 30418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:14.575531 30418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:14.578320 30418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:14.578516 30418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:14.583600 30418 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.126:43947
I20260810 15:25:14.583640 30496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.126:43947 every 8 connection(s)
I20260810 15:25:14.584052 30418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/data/info.pb
I20260810 15:25:14.585144 30498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:14.586787 30498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:14.588215 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 30418
I20260810 15:25:14.588328 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-0/wal/instance
I20260810 15:25:14.588558 4353 external_mini_cluster.cc:1428] Setting key 24d1f48b526b78fa1a024f67609b3ba7
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.125@KRBTEST.COM; defaulting to no policy
I20260810 15:25:14.598747 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.587686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.4.64.126:43947)
0810 15:25:14.587802 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.587807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.587872 (+ 65us) server_negotiation.cc:407] Connection header received
0810 15:25:14.588562 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.588564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.588588 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.588695 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.589390 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.590711 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.592194 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.592519 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.592612 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.595284 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.595292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.595293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.595299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.596996 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.597423 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.597426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.597427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.597513 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.597782 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.597783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.597784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.598017 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.598117 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.598323 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.598489 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":42,"threads_started":1}
I20260810 15:25:14.598750 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.587527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:43947 (local address 127.0.0.1:38858)
0810 15:25:14.587700 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.587769 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.588312 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.588776 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.588779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.589061 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.589262 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.589267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.590815 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.590816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.592092 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.592096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.592644 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.592646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.592775 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.593396 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.593410 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.595176 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.597101 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.597108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.597110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.597314 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.597586 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.597587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.597588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.597703 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.598094 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.598100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.598165 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.598280 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.598556 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":126,"thread_start_us":70,"threads_started":1}
I20260810 15:25:14.599745 30498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:14.600181 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.600067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40366)
0810 15:25:14.600114 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.600161 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:25:14.600327 30438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
I20260810 15:25:14.600768 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.600668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40374)
0810 15:25:14.600719 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.600745 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 15:25:14.600884 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:43237: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab.
Entry for principal kudu/127.4.64.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab.
I20260810 15:25:14.621306 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:43237
--webserver_interface=127.4.64.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab
--principal=kudu/127.4.64.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43947,127.4.64.125:43237,127.4.64.124:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:25:14.650112 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } attempt: 1
I20260810 15:25:14.650679 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.650518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40382)
0810 15:25:14.650587 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.650647 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 15:25:14.651154 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.651054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40388)
0810 15:25:14.651101 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.651131 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 15:25:14.651305 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:43237: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
W20260810 15:25:14.716914 30506 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:25:14.717161 30506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:14.717180 30506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:14.717202 30506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:14.718470 30506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:14.718521 30506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:14.718536 30506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:14.718550 30506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:14.718562 30506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:14.719492 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } attempt: 2
I20260810 15:25:14.720055 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.719907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40402)
0810 15:25:14.719978 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.720024 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 15:25:14.720263 30506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43947,127.4.64.125:43237,127.4.64.124:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.125.keytab
--principal=kudu/127.4.64.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.125:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.64.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:14.720587 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.720383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40404)
0810 15:25:14.720526 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.720559 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 15:25:14.720590 30506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260810 15:25:14.720760 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.125:43237: Network error: Client connection negotiation failed: client connection to 127.4.64.125:43237: connect: Connection refused (error 111)
I20260810 15:25:14.720948 30506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:14.726195 30511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:14.726327 30506 server_base.cc:1034] running on GCE node
W20260810 15:25:14.726328 30514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:14.726480 30512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:14.726784 30506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:14.727114 30506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:14.728310 30506 hybrid_clock.cc:648] HybridClock initialized: now 1786375514728274 us; error 54 us; skew 500 ppm
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:14.731150 30506 init.cc:377] Logged in from keytab as kudu/127.4.64.125@KRBTEST.COM (short username kudu)
I20260810 15:25:14.731729 30506 webserver.cc:460] Webserver started at http://127.4.64.125:46519/ using document root <none> and password file <none>
I20260810 15:25:14.731968 30506 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:14.732038 30506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:14.732203 30506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:14.733304 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data/instance:
uuid: "c6e91230d8a9401d9a3f579f55ee0ecb"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "1831388024a4493a81c8e301619c6204"
server_key_iv: "c7e72c88da6545de402281e387ccac0d"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.733731 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal/instance:
uuid: "c6e91230d8a9401d9a3f579f55ee0ecb"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "1831388024a4493a81c8e301619c6204"
server_key_iv: "c7e72c88da6545de402281e387ccac0d"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.735122 30506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:14.736017 30521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:14.736259 30506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:14.736368 30506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal
uuid: "c6e91230d8a9401d9a3f579f55ee0ecb"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "1831388024a4493a81c8e301619c6204"
server_key_iv: "c7e72c88da6545de402281e387ccac0d"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.736456 30506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:14.758992 30506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:14.761791 30506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:14.761994 30506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:14.766813 30506 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.125:43237
I20260810 15:25:14.766857 30577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.125:43237 every 8 connection(s)
I20260810 15:25:14.767294 30506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/data/info.pb
I20260810 15:25:14.768298 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 30506
I20260810 15:25:14.768429 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-1/wal/instance
I20260810 15:25:14.768518 30578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:14.768716 4353 external_mini_cluster.cc:1428] Setting key 321b12aa0e8e6310abe2c92b4bb6482e
I20260810 15:25:14.770328 30578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.64.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.64.124@KRBTEST.COM" created.
I20260810 15:25:14.783057 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.771103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:43947 (local address 127.0.0.1:38862)
0810 15:25:14.771343 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.771402 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.772165 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.772516 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.772521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.772957 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.773268 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.773274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.774191 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.774192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.775718 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.775722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.776190 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.776192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.776363 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.776973 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.776986 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.778925 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.780887 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.780893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.780896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.781325 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.781769 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.781770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.781771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.782002 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.782382 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.782389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.782582 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.782715 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.782878 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
I20260810 15:25:14.783054 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.771140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.4.64.126:43947)
0810 15:25:14.771227 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.771230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.771534 (+ 304us) server_negotiation.cc:407] Connection header received
0810 15:25:14.772312 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.772313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.772327 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.772390 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.773391 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.774072 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.775825 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.776060 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.776129 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.779078 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.779080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.779081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.779090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.780795 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.781475 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.781477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.781478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.781571 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.782095 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.782096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.782096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.782285 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.782390 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.782783 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.782908 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:25:14.783768 30578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:25:14.792248 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.784082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40416)
0810 15:25:14.784139 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.784151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.784198 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.784520 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.784521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.784678 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.784726 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.784728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.785719 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.785720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.786226 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.786227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.786734 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.786735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.786806 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.787249 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.787254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.788933 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.791040 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.791041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.791041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.791220 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.791465 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.791466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.791466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.791540 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.791870 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.791871 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.791914 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.791982 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.792099 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:25:14.792277 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.784176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.4.64.125:43237)
0810 15:25:14.784379 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.784383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.784401 (+ 18us) server_negotiation.cc:407] Connection header received
0810 15:25:14.784423 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.784424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.784435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.784520 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.784819 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.785546 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.786326 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.786655 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.786739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.789047 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.789057 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.789058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.789065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.790929 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.791321 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.791324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.791325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.791384 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.791611 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.791612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.791612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.791788 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.791884 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.792025 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.792164 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":172,"thread_start_us":116,"threads_started":1}
I20260810 15:25:14.793076 30578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:14.793556 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.793413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45342)
0810 15:25:14.793476 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.793530 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 15:25:14.793776 30523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
I20260810 15:25:14.794433 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.794283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45350)
0810 15:25:14.794367 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.794401 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 15:25:14.794574 30578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:35363: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.64.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab.
Entry for principal kudu/127.4.64.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab.
I20260810 15:25:14.802054 4353 external_mini_cluster.cc:1326] Running /tmp/dist-test-task97bH11/build/release/bin/kudu
/tmp/dist-test-task97bH11/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:35363
--webserver_interface=127.4.64.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab
--principal=kudu/127.4.64.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.64.84:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.64.126:43947,127.4.64.125:43237,127.4.64.124:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:25:14.811980 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } attempt: 3
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:25:14.821033 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.812634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40428)
0810 15:25:14.812722 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.812739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.812817 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.813033 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.813034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.813272 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.813347 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.813349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.814376 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.814377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.814987 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.814989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.815518 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.815518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.815589 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.816152 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.816156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.817840 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.819751 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.819752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.819752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.819937 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.820287 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.820288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.820289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.820365 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.820734 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.820735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.820787 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.820855 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.820957 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 15:25:14.821096 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.812749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.4.64.125:43237)
0810 15:25:14.812839 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.812841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.812876 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:25:14.812911 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.812913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.812929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.812994 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.813448 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.814277 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.815074 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.815436 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.815505 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.817932 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.817933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.817934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.817942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.819647 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.820021 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.820023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.820024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.820100 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.820463 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.820464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.820464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.820614 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.820689 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.820899 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.820986 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 15:25:14.821923 30498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:25:14.822319 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.822203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45352)
0810 15:25:14.822254 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.822297 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:25:14.822690 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.822596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45356)
0810 15:25:14.822626 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.822672 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260810 15:25:14.822784 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:35363: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
I20260810 15:25:14.843853 30578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } attempt: 1
I20260810 15:25:14.844375 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.844230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45368)
0810 15:25:14.844299 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.844342 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 15:25:14.844759 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.844638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45380)
0810 15:25:14.844706 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.844731 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 15:25:14.844851 30578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:35363: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
I20260810 15:25:14.853926 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } attempt: 1
I20260810 15:25:14.854496 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.854350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45390)
0810 15:25:14.854419 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.854465 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 15:25:14.855026 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.854876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45392)
0810 15:25:14.854962 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.854995 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 15:25:14.855175 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:35363: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
W20260810 15:25:14.895133 30587 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:25:14.895380 30587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:25:14.895397 30587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:25:14.895426 30587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:25:14.896651 30587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:25:14.896705 30587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:25:14.896742 30587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:25:14.896755 30587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:25:14.896809 30587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:25:14.898541 30587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.64.84:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.64.126:43947,127.4.64.125:43237,127.4.64.124:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/krb5kdc/kudu_127.4.64.124.keytab
--principal=kudu/127.4.64.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.64.124:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.64.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:25:14.898840 30587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:25:14.899122 30587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:25:14.904634 30596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:25:14.904641 30593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:14.904661 30587 server_base.cc:1034] running on GCE node
W20260810 15:25:14.904641 30592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:25:14.905222 30587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:25:14.905588 30587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:25:14.906826 30587 hybrid_clock.cc:648] HybridClock initialized: now 1786375514906807 us; error 43 us; skew 500 ppm
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:25:14.909775 30587 init.cc:377] Logged in from keytab as kudu/127.4.64.124@KRBTEST.COM (short username kudu)
I20260810 15:25:14.910341 30587 webserver.cc:460] Webserver started at http://127.4.64.124:41183/ using document root <none> and password file <none>
I20260810 15:25:14.910642 30587 fs_manager.cc:359] Metadata directory not provided
I20260810 15:25:14.910725 30587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:25:14.910892 30587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:25:14.912042 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data/instance:
uuid: "fd615f034d304349bcc90353341d5318"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "47c825435606a55c213accca8e9238f5"
server_key_iv: "dc5fa0118e877ca2588c7085f22ff144"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.912432 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal/instance:
uuid: "fd615f034d304349bcc90353341d5318"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "47c825435606a55c213accca8e9238f5"
server_key_iv: "dc5fa0118e877ca2588c7085f22ff144"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.912997 30578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } attempt: 2
I20260810 15:25:14.913408 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.913272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45396)
0810 15:25:14.913338 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.913379 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 15:25:14.913791 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.913692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45408)
0810 15:25:14.913739 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.913764 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 15:25:14.913909 30578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:35363: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
I20260810 15:25:14.914289 30587 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 15:25:14.915197 30605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:25:14.915434 30587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:25:14.915517 30587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data,/tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal
uuid: "fd615f034d304349bcc90353341d5318"
format_stamp: "Formatted at 2026-08-10 15:25:14 on dist-test-slave-9zdj"
server_key: "47c825435606a55c213accca8e9238f5"
server_key_iv: "dc5fa0118e877ca2588c7085f22ff144"
server_key_version: "encryptionkey@0"
I20260810 15:25:14.915575 30587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:25:14.928560 30587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:25:14.930373 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } attempt: 2
I20260810 15:25:14.930891 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.930737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45414)
0810 15:25:14.930812 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.930857 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 15:25:14.930986 30587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:25:14.931161 30587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:25:14.931444 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.931284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45426)
0810 15:25:14.931353 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.931402 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 15:25:14.931568 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.64.124:35363: Network error: Client connection negotiation failed: client connection to 127.4.64.124:35363: connect: Connection refused (error 111)
I20260810 15:25:14.935655 30587 rpc_server.cc:307] RPC server started. Bound to: 127.4.64.124:35363
I20260810 15:25:14.935727 30667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.64.124:35363 every 8 connection(s)
I20260810 15:25:14.936112 30587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/data/info.pb
I20260810 15:25:14.937346 30668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:25:14.938552 4353 external_mini_cluster.cc:1388] Started /tmp/dist-test-task97bH11/build/release/bin/kudu as pid 30587
I20260810 15:25:14.938645 4353 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task97bH11/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786375228510355-4353-0/minicluster-data/master-2/wal/instance
I20260810 15:25:14.938879 4353 external_mini_cluster.cc:1428] Setting key 6de20f697c2c8f760b10e6e0a4b812df
I20260810 15:25:14.940064 30668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:25:14.950656 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.941169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.4.64.126:43947)
0810 15:25:14.941244 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.941246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.941382 (+ 136us) server_negotiation.cc:407] Connection header received
0810 15:25:14.941436 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.941437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.941449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.941514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.942212 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.942708 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.943829 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.944020 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.944078 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.946997 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.946999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.947000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.947011 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.948891 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.949464 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.949465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.949465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.949530 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.949848 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.949850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.949850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.950028 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.950106 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.950458 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.950549 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:25:14.950853 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.941065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.126:43947 (local address 127.0.0.1:38876)
0810 15:25:14.941261 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.941305 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.941385 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.941545 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.941548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.941962 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.942099 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.942105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.942798 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.942799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.943736 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.943738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.944139 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.944140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.944264 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.945039 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.945054 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.946832 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.948988 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.948995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.948998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.949389 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.949597 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.949598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.949598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.949768 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.950101 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.950107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.950283 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.950411 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.950588 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":134,"threads_started":1}
I20260810 15:25:14.951104 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.939526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.4.64.124:35363)
0810 15:25:14.939753 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.939764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.939807 (+ 43us) server_negotiation.cc:407] Connection header received
0810 15:25:14.940412 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.940422 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.940496 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.940671 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.941524 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.942852 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.943653 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.944125 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.944264 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.947428 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.947437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.947438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.947445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.949377 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.950030 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.950033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.950033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.950107 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.950429 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.950430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.950430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.950601 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.950739 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.950913 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.951010 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260810 15:25:14.951610 30668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.126@KRBTEST.COM
I20260810 15:25:14.959329 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.952019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.4.64.125:43237)
0810 15:25:14.952065 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.952067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.952319 (+ 252us) server_negotiation.cc:407] Connection header received
0810 15:25:14.952406 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.952407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.952417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.952467 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.952790 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.953276 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.954134 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.954376 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.954457 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.956289 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.956290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.956290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.956297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.958121 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.958463 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.958465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.958465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.958529 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.958716 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.958717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.958717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.958871 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.958924 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.959131 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.959221 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:25:14.959342 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.951937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.125:43237 (local address 127.0.0.1:40442)
0810 15:25:14.952238 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.952249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.952351 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.952468 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.952468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.952639 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.952708 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.952710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.953367 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.953368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.954036 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.954038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.954453 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.954454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.954525 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.954955 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.954959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.956197 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.958217 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.958219 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.958219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.958388 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.958600 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.958601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.958601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.958653 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.958960 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.958961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.959014 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.959091 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.959188 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260810 15:25:14.960107 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.952307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.4.64.126:43947)
0810 15:25:14.952353 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.952355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.952365 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:25:14.952389 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.952390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.952400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.952453 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.952961 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.953485 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.954340 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.954543 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.954629 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.956886 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.956888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.956888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.956895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.958675 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.959106 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.959107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.959108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.959169 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.959454 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.959455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.959455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.959605 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.959865 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.959904 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.960005 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:25:14.960085 30668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.124@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
Aug 10 15:25:14 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.64.125@KRBTEST.COM
I20260810 15:25:14.967101 30672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.960367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45446)
0810 15:25:14.960403 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:14.960415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:14.960459 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:14.960567 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:14.960567 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:14.960742 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:14.960780 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.960781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.961444 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.961445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.962004 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:14.962006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.962383 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.962383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:14.962432 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.962869 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:14.962874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:14.964175 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:14.966012 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.966013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.966013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.966188 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.966414 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:14.966415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:14.966415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:14.966472 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:14.966734 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:14.966735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:14.966776 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 15:25:14.966879 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 15:25:14.966988 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 15:25:14.967113 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.960402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.4.64.124:35363)
0810 15:25:14.960436 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.960438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.960445 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:25:14.960514 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.960515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.960523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.960566 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.960869 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.961349 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.962084 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.962305 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.962385 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.964264 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.964265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.964265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.964272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.965900 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.966279 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.966280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.966280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.966333 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.966544 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.966545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.966546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.966665 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.966742 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.966905 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.967038 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":20}
I20260810 15:25:14.967456 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:14.960715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.4.64.125:43237)
0810 15:25:14.960772 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:14.960775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:14.960789 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:25:14.960809 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:14.960810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:14.960818 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:14.960866 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:14.961330 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.961883 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.962537 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:14.962758 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:14.962840 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:14.964975 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:14.964976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:14.964977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:14.964985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:14.966328 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.966677 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.966679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.966679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.966738 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:14.966946 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:14.966947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:14.966947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:14.967063 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:14.967244 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:14.967281 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:25:14.967356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:25:14.969141 4353 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:25:14.969179 30668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: Bootstrap starting.
I20260810 15:25:14.970171 30668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:14.970476 30668 log.cc:824] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:14.970286 30465 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "37d542c072584a3381900ff0b200d96a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38882
I20260810 15:25:14.971369 30668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: No bootstrap required, opened a new log
I20260810 15:25:14.972790 30668 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:14.973006 30668 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd615f034d304349bcc90353341d5318, State: Initialized, Role: FOLLOWER
I20260810 15:25:14.973160 30668 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:14.973496 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:14.973541 30668 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:14.973644 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:25:14.975464 30688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:25:14.975525 30688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:25:14.975906 30587 master_runner.cc:186] Error getting master registration for 127.4.64.126:43947: OK, instance_id { permanent_uuid: "37d542c072584a3381900ff0b200d96a" instance_seqno: 1786375514582126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:25:14.978888 30587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:25:15.005160 30578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } attempt: 3
Aug 10 15:25:15 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.125@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:25:15.013458 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.005671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45448)
0810 15:25:15.005734 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:15.005750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:15.005828 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:15.006007 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:15.006008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:15.006196 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:15.006248 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:15.006249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.006930 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.006931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:15.007428 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:15.007430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.007795 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.007797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:15.007854 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.008501 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:15.008506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:15.010291 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:15.012270 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:15.012271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:15.012271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:15.012459 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:15.012698 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:15.012699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:15.012699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:15.012762 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:15.013086 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:15.013087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:15.013138 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 15:25:15.013230 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:25:15.013346 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:25:15.013453 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.005783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.4.64.124:35363)
0810 15:25:15.005830 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.005832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.005843 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:25:15.005908 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.005909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.005919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.005999 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.006346 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.006837 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.007518 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.007715 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.007800 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.010393 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:15.010394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:15.010394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:15.010403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:15.012176 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.012538 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.012539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.012540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.012602 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.012833 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.012834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.012834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.013013 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:15.013095 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:15.013269 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 15:25:15.013353 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:25:15.015158 30578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: Bootstrap starting.
I20260810 15:25:15.015938 30578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:15.016285 30578 log.cc:824] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:15.017102 30578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: No bootstrap required, opened a new log
I20260810 15:25:15.018924 30578 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:15.019080 30578 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6e91230d8a9401d9a3f579f55ee0ecb, State: Initialized, Role: FOLLOWER
I20260810 15:25:15.019186 30578 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:15.019564 30578 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:15.019529 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.019686 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:25:15.021524 30703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:25:15.021622 30703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:25:15.021862 30506 master_runner.cc:186] Error getting master registration for 127.4.64.126:43947: OK, instance_id { permanent_uuid: "37d542c072584a3381900ff0b200d96a" instance_seqno: 1786375514582126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:25:15.024362 30506 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:25:15.030691 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } attempt: 3
Aug 10 15:25:15 dist-test-slave-9zdj krb5kdc[30402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375514, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.64.126@KRBTEST.COM for kudu/127.4.64.124@KRBTEST.COM
I20260810 15:25:15.039863 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.031063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.64.124:35363 (local address 127.0.0.1:45462)
0810 15:25:15.031188 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 15:25:15.031202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:25:15.031248 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:25:15.031589 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:25:15.031590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:25:15.031846 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:25:15.031905 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:15.031908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.032882 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.032884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:15.033763 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:25:15.033766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.034264 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.034265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:25:15.034321 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.034840 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:25:15.034847 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:25:15.036546 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:25:15.038607 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:15.038608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:15.038609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:15.038867 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:15.039139 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:25:15.039140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:25:15.039140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:25:15.039210 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:25:15.039510 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:25:15.039511 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:25:15.039560 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:25:15.039622 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:25:15.039749 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 15:25:15.040452 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.031335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.4.64.124:35363)
0810 15:25:15.031378 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.031380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.031388 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:25:15.031416 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.031417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.031427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.031475 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.032005 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.032765 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.033862 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.034136 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.034199 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.036680 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:15.036681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:15.036682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:15.036691 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:15.038518 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.038984 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.038986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.038986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.039053 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.039277 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.039278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.039278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.039417 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:15.039484 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:15.040267 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 15:25:15.040357 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:25:15.042687 30498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: Bootstrap starting.
I20260810 15:25:15.043491 30498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: Neither blocks nor log segments found. Creating new log.
I20260810 15:25:15.043823 30498 log.cc:824] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:25:15.044818 30498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: No bootstrap required, opened a new log
I20260810 15:25:15.046298 30498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:15.046443 30498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d542c072584a3381900ff0b200d96a, State: Initialized, Role: FOLLOWER
I20260810 15:25:15.046522 30498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:15.046905 30465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 15:25:15.046988 30465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:15.047852 30465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:15.048126 30465 leader_election.cc:288] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [CANDIDATE]: Term 1 election: Requested vote from peers c6e91230d8a9401d9a3f579f55ee0ecb (127.4.64.125:43237), fd615f034d304349bcc90353341d5318 (127.4.64.124:35363)
I20260810 15:25:15.046875 30498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:25:15.048597 30550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37d542c072584a3381900ff0b200d96a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb"
I20260810 15:25:15.048746 30550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:15.049742 30550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d542c072584a3381900ff0b200d96a in term 1.
I20260810 15:25:15.050982 30438 leader_election.cc:302] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37d542c072584a3381900ff0b200d96a, c6e91230d8a9401d9a3f579f55ee0ecb; no voters:
I20260810 15:25:15.051460 30636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37d542c072584a3381900ff0b200d96a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd615f034d304349bcc90353341d5318"
I20260810 15:25:15.051577 30636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:25:15.052471 30636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37d542c072584a3381900ff0b200d96a in term 1.
I20260810 15:25:15.053045 30726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:25:15.053269 30726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [term 1 LEADER]: Becoming Leader. State: Replica: 37d542c072584a3381900ff0b200d96a, State: Running, Role: LEADER
I20260810 15:25:15.053418 30726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } }
I20260810 15:25:15.054355 30418 master_runner.cc:263] No leader master found from master 37d542c072584a3381900ff0b200d96a
I20260810 15:25:15.054400 30418 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:25:15.054734 30726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d542c072584a3381900ff0b200d96a. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.046856 30706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
W20260810 15:25:15.057443 30718 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:25:15.057871 30718 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:25:15.057812 30706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:15.058009 30726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:15.058904 30732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:25:15.059085 30732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:25:15.070340 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.054185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.4.64.126:43947)
0810 15:25:15.054229 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.054237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.059668 (+ 5431us) server_negotiation.cc:407] Connection header received
0810 15:25:15.059772 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.059774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.059785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.059816 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.060632 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.061379 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.062499 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.062854 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.062902 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.064933 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:15.064934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:15.064935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:15.064943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:15.067054 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.067692 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.067693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.067694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.067759 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.069683 (+ 1924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.069685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.069685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.069839 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:15.069915 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:15.070136 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 15:25:15.070237 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:25:15.077394 30636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [term 1 FOLLOWER]: Refusing update from remote peer 37d542c072584a3381900ff0b200d96a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:25:15.078972 30726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:25:15.079340 30550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [term 1 FOLLOWER]: Refusing update from remote peer 37d542c072584a3381900ff0b200d96a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 15:25:15.079424 30440 tablet.cc:2369] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:25:15.079814 30726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:25:15.081214 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d542c072584a3381900ff0b200d96a. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.081319 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:15.082616 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d542c072584a3381900ff0b200d96a. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.082727 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:15.083545 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.056253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.4.64.125:43237)
0810 15:25:15.056293 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.056296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.065838 (+ 9542us) server_negotiation.cc:407] Connection header received
0810 15:25:15.065918 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.065919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.065932 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.065991 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.067837 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.068504 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.071454 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.071786 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.071830 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.073666 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:15.073667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:15.073667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:15.073676 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:15.075684 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.076124 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.076125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.076125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.076188 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.076400 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.076401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.076401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.076540 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:15.076635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:15.083331 (+ 6696us) server_negotiation.cc:299] Negotiation successful
0810 15:25:15.083442 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:25:15.087381 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:15.059364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.4.64.124:35363)
0810 15:25:15.059411 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.059413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.065795 (+ 6382us) server_negotiation.cc:407] Connection header received
0810 15:25:15.066123 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.066124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.066136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.066178 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.067383 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.068136 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.073945 (+ 5809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:15.074273 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:15.074323 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:15.084082 (+ 9759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:25:15.084084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:25:15.084084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:25:15.084095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:25:15.085984 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.086454 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.086456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.086456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.086523 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:25:15.086751 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:25:15.086752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:25:15.086752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:25:15.086897 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:25:15.086998 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:15.087197 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 15:25:15.087281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:25:15.088918 30692 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.088907 30706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.089015 30692 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:15.089023 30706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:15.090276 30741 mvcc.cc:205] Tried to move back new op lower bound from 7316994109752459264 to 7316994109663137792. Current Snapshot: MvccSnapshot[applied={T|T < 7316994109752459264}]
I20260810 15:25:15.090971 30727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.091085 30727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a [sys.catalog]: This master's current role is: LEADER
I20260810 15:25:15.091351 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37d542c072584a3381900ff0b200d96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d542c072584a3381900ff0b200d96a" member_type: VOTER last_known_addr { host: "127.4.64.126" port: 43947 } } peers { permanent_uuid: "c6e91230d8a9401d9a3f579f55ee0ecb" member_type: VOTER last_known_addr { host: "127.4.64.125" port: 43237 } } peers { permanent_uuid: "fd615f034d304349bcc90353341d5318" member_type: VOTER last_known_addr { host: "127.4.64.124" port: 35363 } } }
I20260810 15:25:15.091452 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:25:15.091863 30736 mvcc.cc:205] Tried to move back new op lower bound from 7316994109752459264 to 7316994109663137792. Current Snapshot: MvccSnapshot[applied={T|T < 7316994109752459264}]
I20260810 15:25:15.098238 30732 catalog_manager.cc:1347] Generated new cluster ID: 3c46276c70ba460aa6de1e1309f0b049
I20260810 15:25:15.098330 30732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:25:15.112087 30732 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:25:15.112965 30732 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:25:15.132244 30732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d542c072584a3381900ff0b200d96a: Generated new TSK 0
I20260810 15:25:15.134338 30732 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:25:15.171453 30501 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.169282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.4.64.126:43947)
0810 15:25:15.169331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.169333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.170674 (+ 1341us) server_negotiation.cc:407] Connection header received
0810 15:25:15.170806 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.170807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.170820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.170854 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.171371 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38896: BlockingRecv error: recv got EOF from 127.0.0.1:38896 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260810 15:25:15.172524 30671 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.169532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.4.64.124:35363)
0810 15:25:15.169574 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.169576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.171902 (+ 2326us) server_negotiation.cc:407] Connection header received
0810 15:25:15.172015 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.172016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.172028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.172059 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.172454 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45472: BlockingRecv error: recv got EOF from 127.0.0.1:45472 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:25:15.173512 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.170239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.4.64.125:43237)
0810 15:25:15.170279 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.170281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.172930 (+ 2649us) server_negotiation.cc:407] Connection header received
0810 15:25:15.173032 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.173033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.173045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.173075 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.173417 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40462: BlockingRecv error: recv got EOF from 127.0.0.1:40462 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:25:15.176769 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.175932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.4.64.125:43237)
0810 15:25:15.175968 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.175969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.176181 (+ 212us) server_negotiation.cc:407] Connection header received
0810 15:25:15.176290 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.176291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.176310 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.176352 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.176719 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40464: BlockingRecv error: recv got EOF from 127.0.0.1:40464 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:25:15.180111 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.179297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.4.64.125:43237)
0810 15:25:15.179332 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.179333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.179573 (+ 240us) server_negotiation.cc:407] Connection header received
0810 15:25:15.179673 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.179674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.179685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.179714 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.180061 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40474: BlockingRecv error: recv got EOF from 127.0.0.1:40474 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:25:15.184361 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.183503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.4.64.125:43237)
0810 15:25:15.183541 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.183543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.183799 (+ 256us) server_negotiation.cc:407] Connection header received
0810 15:25:15.183905 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.183906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.183924 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.183955 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.184311 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40476: BlockingRecv error: recv got EOF from 127.0.0.1:40476 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:25:15.190862 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.189938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.4.64.125:43237)
0810 15:25:15.189979 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.189981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.190241 (+ 260us) server_negotiation.cc:407] Connection header received
0810 15:25:15.190352 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.190353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.190365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.190396 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.190777 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40488: BlockingRecv error: recv got EOF from 127.0.0.1:40488 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:25:15.201431 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.200558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.4.64.125:43237)
0810 15:25:15.200613 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.200615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.200793 (+ 178us) server_negotiation.cc:407] Connection header received
0810 15:25:15.200851 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.200852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.200866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.200937 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.201366 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40504: BlockingRecv error: recv got EOF from 127.0.0.1:40504 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 15:25:15.219650 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.218695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.4.64.125:43237)
0810 15:25:15.218739 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.218742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.218898 (+ 156us) server_negotiation.cc:407] Connection header received
0810 15:25:15.219051 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.219052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.219093 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.219167 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.219582 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40506: BlockingRecv error: recv got EOF from 127.0.0.1:40506 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 15:25:15.253849 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.252881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.4.64.125:43237)
0810 15:25:15.252962 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.252965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.253089 (+ 124us) server_negotiation.cc:407] Connection header received
0810 15:25:15.253165 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.253166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.253181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.253264 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.253771 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40512: BlockingRecv error: recv got EOF from 127.0.0.1:40512 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 15:25:15.320595 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.319573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.4.64.125:43237)
0810 15:25:15.319654 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.319657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.319788 (+ 131us) server_negotiation.cc:407] Connection header received
0810 15:25:15.319912 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.319913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.319925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.319980 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.320510 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40524: BlockingRecv error: recv got EOF from 127.0.0.1:40524 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 15:25:15.450711 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.449903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40540 (local address 127.4.64.125:43237)
0810 15:25:15.449982 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.449985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.450061 (+ 76us) server_negotiation.cc:407] Connection header received
0810 15:25:15.450131 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.450132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.450142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.450264 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.450645 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40540: BlockingRecv error: recv got EOF from 127.0.0.1:40540 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 15:25:15.709077 30584 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:25:15.708257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.4.64.125:43237)
0810 15:25:15.708355 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:15.708357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:15.708391 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:25:15.708493 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:15.708494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:15.708505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:15.708569 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:25:15.708983 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40556: BlockingRecv error: recv got EOF from 127.0.0.1:40556 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 15:25:15.976119 30688 catalog_manager.cc:1259] Loaded cluster ID: 3c46276c70ba460aa6de1e1309f0b049
I20260810 15:25:15.976275 30688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: loading cluster ID for follower catalog manager: success
I20260810 15:25:15.977087 30688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: acquiring CA information for follower catalog manager: success
I20260810 15:25:15.977344 30688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd615f034d304349bcc90353341d5318: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:25:16.022156 30703 catalog_manager.cc:1259] Loaded cluster ID: 3c46276c70ba460aa6de1e1309f0b049
I20260810 15:25:16.022243 30703 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: loading cluster ID for follower catalog manager: success
I20260810 15:25:16.023130 30703 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: acquiring CA information for follower catalog manager: success
I20260810 15:25:16.023486 30703 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6e91230d8a9401d9a3f579f55ee0ecb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:25:16.225435 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:16.222382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.4.64.125:43237)
0810 15:25:16.222516 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:16.222518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:16.222621 (+ 103us) server_negotiation.cc:407] Connection header received
0810 15:25:16.222777 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:16.222778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:16.222832 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:16.222924 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:16.223102 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.223637 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.224498 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.224717 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.224798 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:16.225005 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:16.225173 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:16.225270 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:16.225294 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 15:25:16.225321 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260810 15:25:16.230330 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:16.227402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.4.64.124:35363)
0810 15:25:16.227529 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:16.227531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:16.227593 (+ 62us) server_negotiation.cc:407] Connection header received
0810 15:25:16.227708 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:16.227709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:16.227757 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:16.227851 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:16.227954 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.228657 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.229435 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.229693 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.229778 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:16.229942 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:16.230059 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:16.230158 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:16.230200 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:25:16.230224 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260810 15:25:16.235313 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:16.232173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.4.64.124:35363)
0810 15:25:16.232214 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:16.232216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:16.232464 (+ 248us) server_negotiation.cc:407] Connection header received
0810 15:25:16.232542 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:16.232542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:16.232553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:16.232612 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:16.232769 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.233239 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.234430 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.234691 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.234724 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:16.234920 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:16.234955 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:16.234986 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:16.235191 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 15:25:16.235214 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:25:16.235445 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:16.232054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.4.64.126:43947)
0810 15:25:16.232242 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:16.232244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:16.232252 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:25:16.232297 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:16.232297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:16.232349 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:16.232446 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:16.232633 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.233433 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.234494 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.234799 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.234872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:16.235083 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:16.235206 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:16.235267 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:16.235323 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:25:16.235353 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":51,"threads_started":1}
I20260810 15:25:16.235519 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:25:16.232127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.4.64.125:43237)
0810 15:25:16.232174 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:25:16.232176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:25:16.232425 (+ 249us) server_negotiation.cc:407] Connection header received
0810 15:25:16.232469 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:25:16.232470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:25:16.232484 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:25:16.232537 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:25:16.232724 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.233443 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.234557 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:25:16.234885 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:25:16.234951 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:25:16.235092 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:25:16.235125 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:25:16.235174 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:25:16.235403 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 15:25:16.235430 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:25:16.237665 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 30418
I20260810 15:25:16.247249 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 30506
I20260810 15:25:16.255393 4353 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task97bH11/build/release/bin/kudu with pid 30587
2026-08-10T15:25:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3179 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6124 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287756 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST